Augmented reality enhancements for dental practitioners

ABSTRACT

A system comprises a scanner, an augmented reality (AR) display and a computing device. The scanner generates intraoral images of a dental arch and the AR display generates additional image data representative of a view from a wearer of the AR display. The computing device receives the intraoral images from the intraoral scanner, generates a virtual three-dimensional model of at least a portion of the dental arch from the intraoral images, receives the additional image data from the AR display, determines, from the additional image data, a region of the view that is outside of the dental arch, generates a visual overlay comprising the virtual three-dimensional model, and sends the visual overlay to the AR display. The AR display displays the visual overlay such that the virtual three-dimensional model is shown in the region of the view that is outside of the dental arch.

RELATED APPLICATIONS

This patent application claims the benefit under 35 U.S.C. § 119(e) ofU.S. Provisional Application No. 62/435,565, filed Dec. 16, 2016, whichis incorporated by reference herein.

TECHNICAL FIELD

Embodiments of the present invention relate to the field of dentistryand, in particular, to a system and method for providing augmentedreality enhancements for dental practitioners.

BACKGROUND

Augmented reality devices may provide additional information to users ofthe devices in the context of the surrounding real world environment.For example, an augmented reality device may provide audio, video,graphic, or other information to a user to supplement the informationavailable in the real world environment.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are illustrated by way of example,and not by way of limitation, in the figures of the accompanyingdrawings.

FIG. 1A illustrates one embodiment of an augmented reality system forenhancing the dental practice of dental practitioners, in accordancewith an embodiment.

FIG. 1B illustrates one embodiment of an augmented reality processingmodule, in accordance with an embodiment.

FIG. 2 illustrates a flow diagram for a method of determining areas ofinterest by an augmented reality device based on comparison to previousimage data, in accordance with an embodiment.

FIG. 3 illustrates a flow diagram for a method of registering image datafrom an augmented reality device to a three dimensional model, inaccordance with an embodiment.

FIG. 4 illustrates a flow diagram for a method of determiningdifferences between a dental arch as depicted in image data from anaugmented reality device and the dental arch as depicted in previousimage data, in accordance with an embodiment.

FIG. 5 illustrates a flow diagram for a method of tracking progress ofan orthodontic treatment plan using image data from an augmented realitydevice, in accordance with an embodiment.

FIG. 6 illustrates a flow diagram for a method of augmenting the view ofa patient's mouth through an augmented reality display based on aclinical history of the patient, in accordance with an embodiment.

FIG. 7 illustrates a flow diagram for a method of augmenting the view ofa patient's mouth through an augmented reality display, in accordancewith an embodiment.

FIG. 8A illustrates a view of an example augmented reality displayshowing areas of interest, in accordance with an embodiment.

FIG. 8B illustrates a view of an example augmented reality displayshowing areas of interest, in accordance with an embodiment.

FIG. 9 illustrates a flow diagram for a method of determining areas ofinterest by an augmented reality device, in accordance with anembodiment.

FIG. 10 illustrates a flow diagram for a method of processing image dataof a dental arch from an augmented reality device based on machinelearning profiles of dental conditions, in accordance with anembodiment.

FIG. 11 illustrates a flow diagram for a method of processing image dataof a dental arch from an augmented reality device to identify toothwear, in accordance with an embodiment.

FIG. 12A illustrates a portion of an example augmented reality displayshowing areas of interest, in accordance with an embodiment.

FIG. 12B illustrates a portion of an example augmented reality displayshowing areas of interest, in accordance with an embodiment.

FIG. 13 illustrates a flow diagram for a method of enhancing a view of apatient's mouth as viewed through an augmented reality device, inaccordance with an embodiment.

FIG. 14 illustrates a flow diagram for a method of providing a visualoverlay of a patient's mouth during a dental procedure to augment thedental procedure, in accordance with an embodiment.

FIG. 15A illustrates a portion of an example augmented reality displayshowing areas of interest related to grinding a tooth, in accordancewith an embodiment.

FIG. 15B illustrates a portion of an example augmented reality displayshowing areas of interest related to grinding a tooth, in accordancewith an embodiment.

FIG. 16 illustrates a view of an example augmented reality displayshowing areas of interest related to an insertion path for a dentalimplant, in accordance with an embodiment.

FIG. 17A illustrates a portion of an example augmented reality displayshowing areas of interest that identify an occlusion map, in accordancewith an embodiment.

FIG. 17B illustrates a view of an example augmented reality displayshowing actual teeth movement vs. target teeth movement for a treatmentplan, in accordance with an embodiment.

FIG. 18 illustrates a flow diagram for a method of determining areas ofinterest by an augmented reality device, in accordance with anembodiment.

FIG. 19 illustrates a flow diagram for a method of providing a visualoverlay in an image on an augmented reality device that providesinformation about a procedure to grind a tooth, in accordance with anembodiment.

FIG. 20 illustrates a flow diagram for a method of providing a visualoverlay with information that augments use of a dental tool, inaccordance with an embodiment.

FIG. 21 illustrates a flow diagram for a method of facilitatingplacement of attachments on a patient's teeth using an augmented realitydevice, in accordance with an embodiment.

FIG. 22 illustrates a flow diagram for a method of facilitating anintraoral scan session using an augmented reality device, in accordancewith an embodiment.

FIG. 23 illustrates a flow diagram for a method of using an augmentedreality display for an intraoral scanner, in accordance with anembodiment.

FIG. 24A illustrates a flow diagram for another method of using anaugmented reality display for an intraoral scanner, in accordance withan embodiment.

FIG. 24B illustrates a virtual display for an intraoral scan applicationthat is displayed on an AR display, in accordance with an embodiment.

FIG. 25A illustrates a flow diagram for a method of using an augmentedreality display and an intraoral scanner to provide a zoomed in view ofa dental procedure, in accordance with an embodiment.

FIG. 25B illustrates a dental practitioner operating on a patient, inaccordance with an embodiment.

FIG. 26 illustrates a flow diagram for a method of generating a modelfor a dental arch from images captured by an image capture deviceassociated with an augmented reality display, in accordance with anembodiment.

FIG. 27 illustrates a flow diagram for a method of attaching audio notesto image data from an image capture device associated with an augmentedreality display, in accordance with an embodiment.

FIG. 28 illustrates selected images from a set of images generated by animage capture device associated with an augmented reality display, inaccordance with an embodiment.

FIG. 29 illustrates additional selected images from a set of imagesgenerated by an image capture device associated with an augmentedreality display, in accordance with an embodiment.

FIG. 30 illustrates a block diagram of an example computing device, inaccordance with embodiments of the present invention.

DETAILED DESCRIPTION

Described herein are methods and apparatuses for providing augmentedreality (AR) enhancements to dentists, orthodontists, dental hygienists,or other dental practitioners. Also described is an intraoral scannerthat includes an AR display. An AR system (also referred to herein as anAR device) may provide real-time information to a dental practitionerbased on an analysis of the mouth and/or dental arch of a patient asviewed through an AR display. For example, the AR system may provideinformation about a dental arch based on images captured of the patientby the AR system. The AR system may also provide additional informationbased on a comparison of images captured by the AR system and previousdata recorded for the patient. For example, previous images, scans,models, clinical data or other patient history may be compared to theimages captured by the AR system, and a result of the comparison may beprovided to the dental practitioner as a visual overlay on thereal-world scene viewed by the dental practitioner through an AR displayof the AR system. Previous data about the patient may also be providedin the visual overlay. Additionally, image data from the AR system maybe used to facilitate dental procedures such as drilling, grinding of atooth, placement of an attachment on a tooth, placement of a bracket ona tooth (e.g., a bracket placed in the middle of the crown of a tooth),placement of other objects in pre-defined or automatically identifiedpositions, intraoral scanning, and so on. The AR system may updateinformation provided to a dental practitioner or provide feedback to thedental practitioner in real time or near real time during the course ofthe dental practitioner interacting with the patient.

In some embodiments, an AR system may provide information to the dentalpractitioner based on analysis of image data without using previousinformation about the patient. For example, the AR system may analyze animage or stream of images of a patient's oral cavity and dental arch anddetermine an area of interest present in the image data. The AR systemmay determine if one or more teeth in an image indicate excessive wear,plaque, deposits, cracks, cavities, or other characteristics of interestto dental practitioners. The areas of interest may be determined basedon processing an image of a dental arch or tooth taken by the AR systemusing one or more dental condition profiles in a data store. In someembodiments, the AR system may analyze an image of a tooth, multipleteeth, or a dental arch using dental condition profiles generated usingmachine learning techniques and training data of previous images ofteeth.

After the AR system determines one or more areas of interest, the ARdisplay may then display real world data to a dental practitioner alongwith a visual overlay highlighting the areas of interest to the dentalpractitioner. In an example, the AR display may include lenses throughwhich a wearer views the physical world, and the visual overlay may beprojected onto the lenses. Alternatively, the visual overlay may beprojected directly onto a wearer's eyes. For example, a tooth may behighlighted in a different color, circled, or otherwise indicated ashaving a characteristic in a visual overlay displayed by the AR system.In some embodiments, the AR system may provide different indicators fordifferent characteristics or dental conditions. In some embodiments, anarea of interest may be highlighted, and a reason for the area ofinterest may be output in another portion of the display of the ARsystem or may be output in another manner, such as audio. In someembodiments, the AR system may also enhance a live view of the patient,such as by providing light enhancements that improve viewing of thepatient or providing a zoomed in image of a portion of a patient'smouth.

In some embodiments, the AR system may provide information to the dentalpractitioner based on analysis of the patient and/or in view of previouspatient data. For example, the AR system may compare images or modelsfrom a previous visit to current images of the patient's dental arch.The AR system may then determine one or more areas of interest based onthe comparison. For example, the AR system may identify changes since alast scan, analysis of wear over time, feedback on orthodontictreatment, or other analysis of changes. The AR system may then mark thechanges on a display of the AR system. In some embodiments, the ARsystem may also superimpose previous patient data on a display. Forexample, the AR system may show a previous scan or previous dental archsuperimposed onto a display.

In some embodiments, the AR system may provide interactive feedback orother updated information to the dental practitioner based on aninteraction with the patient. For example, the feedback may be providedduring an intra-oral treatment such as a dental procedure. In someembodiments, the AR system may output to a display of the AR systemrecommended steps to take during an implant procedure, drillingprocedure, grinding procedure, etc. For example, the AR system may showwhere to remove material for an insertion path, potential undercuts ofneighboring teeth, placement of a hole for an implant, drilling depth,drilling direction, or the like. Similarly, the AR system may provide anindication of material to remove during interproximal reduction. In someembodiments, the AR system may provide feedback regarding placement ofan attachment on a tooth. In some embodiments, the AR system maysuperimpose an occlusion map onto the patient's teeth in a display ofthe AR system. The AR system may also update a superimposed occlusionmap if it changes while a dental practitioner is performing a dentalprocedure. An AR system may also provide feedback based on otherinformation or analysis performed on images or other data received abouta patient.

Embodiments provide significant advantages over traditional techniquesfor dentistry and orthodontics, and can improve every aspect of a dentalpractice. Dental hygienists can use an AR system as described herein tobetter interact with a patient and identify potential dental issues thata dental hygienist is qualified to address, such as gum swelling orplaque caused by poor dental hygiene. The AR system may automaticallyprocess image data from the image capture device to identify, forexample, tooth wear, gum swelling, gum discoloration, plaque, etc. andcall these dental conditions to the attention of the dental hygienist.Similarly, a dentist may use an AR system that provides real-timefeedback as described herein to improve his or her accuracy inperforming intraoral procedures such as drilling a tooth, grinding atooth, placing an attachment on a tooth, placing an implant, and so on.The AR system also presents information to a dental practitioner whilethe dental practitioner views a patient, and may reduce or eliminate aneed for the dental practitioner to look away from the patient to acomputer screen or chart. Additionally, an orthodontist may use an ARsystem as described herein to improve his analysis of how an orthodontictreatment plan is progressing, to improve performance of intraoralprocedures, and so on. Embodiments therefore improve the efficiency ofinterfacing with patients, the accuracy of dental procedures and theidentification of dental conditions. For example, embodiments enable adental practitioner to work while looking exclusively at the patient'sjaws, without any reason to turn his or her head toward a screen ormonitor (e.g., of a computing device for an intraoral scanner).

In some embodiments, an intraoral scanner uses an AR display as aprimary or secondary display for controlling an intraoral scanningprocedure. The AR display may be worn by a dental practitioner that usesthe intraoral scanner to image a patient's dental arch and generate avirtual three-dimensional model of that dental arch. The AR display mayprovide a two-dimensional (2-D) or three-dimensional (3-D) menu ofoptions for controlling the intraoral scan procedure. Additionally, theAR display may be used to provide a zoomed in view of a region of thedental arch being scanned. Additionally, the AR display may be used toprovide a virtual overlay of a virtual 3-D model of the dental archbased on images generated by the intraoral scanner during an intraoralscan procedure.

During an intraoral scan procedure (also referred to as a scan session),a user (e.g., a dental practitioner) of an intraoral scanner maygenerate multiple different images (also referred to as scans or medicalimages) of a dental site, model of a dental site, or other object. Theimages may be discrete images (e.g., point-and-shoot images) or framesfrom a video (e.g., a continuous scan). Existing medical scanningsolutions frequently involve the user holding the scanner to engage thepatient for scanning, disengaging from the patient to address a medicalscan application executing on a computing device, then reengaging withthe patient to continue the scanning process, again disengaging from thepatient to address the medical scan application, and repeating untilcompletion of a scanning session. Such processes can be quite cumbersomeand inefficient. Moreover, medical scanning devices generally lack theability to both generate medical images and then manipulate thosemedical images or representations thereof on a display of a computingdevice.

Embodiments of the present invention enable a user to perform operations(such as to control or navigate a user interface and/or to manipulateintraoral images or a representation generated from intraoral images)while still engaged with a patient that in previous systems could onlybe performed by disengaging from the patient and interacting with acomputing device running an intraoral scan application. The dentalpractitioner may see a menu for the intraoral scan application overlaidon a field of view of the dental practitioner while the dentalpractitioner remains focused on the patient. The ability to perform suchoperations while still engaged with the patient can improve theefficiency of a workflow for scanning a patient or performing otheroperations. This will save the dental practitioner time duringtreatment. It also allows the dental practitioner to see the effects ofhis or her work and enable him or her to correct and adjust treatment inreal time as appropriate.

In some embodiments, an image capture device of an AR display may beused to generate multiple images of a patient's face. The image capturedevice may generate a stream of images, and processing logic may analyzethe stream of images to select a subset of those images. The selectedsubset of images may then be saved and used to generate a modelassociated with a dental arch or jaw, such as an articulation model ofthe patient's jaw. Additionally, a dental practitioner wearing the ARdisplay may generate voice notes and append those voice notes to imagestaken by the image capture device of the AR display.

Embodiments described herein are discussed with reference to an ARsystem. An AR system is a device that enables a live direct or indirectview of a physical, real-world environment and that augments the view ofthe physical real-world environment by computer generated sensory inputsuch as sound, video, or graphics. An AR system may include an ARdisplay that includes glasses or other lenses that have one or morecameras attached to capture images of a patient. The AR display may alsohave a projector that projects images onto the glasses or lenses toprovide a visual overlay to a dental practitioner. The visual overlay issuperimposed over the real world image that the dental practitioner seesthrough the glasses or lenses. Some embodiments herein are describedwith reference to an AR display that is worn by a dental practitioner,such as AR glasses, AR goggles, or an AR headset. While some embodimentsdescribed herein are discussed with reference to a worn AR display, itshould be understood that embodiments also apply to AR system that useother types of displays. For example, embodiments may apply to acomputing device having a screen showing live images captured of apatient and overlay information to enhance the experience of the dentalpractitioner viewing the screen.

Additionally, it should be understood that embodiments described withreference to an AR system also apply to a virtual reality (VR) system. AVR system is similar to an AR system, except that an AR system allows awearer or viewer to see an augmented version of the real world, while aVR system provides a purely simulated environment. A VR systemartificially creates sensory experiences that can include sight, touch,sound, and/or other senses, and presents these sensory experiences ontoa VR display. Any reference made herein to any type of AR system and/orAR display applies equally to a VR system and/or VR display.

FIG. 1A illustrates one embodiment of an AR system 100 for providingaugmented reality enhancements to a dental practitioner. In oneembodiment, the AR system 100 includes a computing device 105, an ARdisplay 150, an image capture device 160, and a data store 110. In someembodiments, the image capture device 160 is a component of the ARdisplay 150. In some embodiments, multiple components shown in FIG. 1Amay be integrated into a device that houses the AR display 150. Forexample, the computing device 105 and image capture device 160 may beintegrated into glasses or a headset to be worn by a dentalpractitioner. In some embodiments, the computing device 105 may beseparate from the AR display 150, but connected through either a wiredor wireless connection to a processing device in the AR display 150.Additionally, the data store 110 may be attached to the AR display 150,may be directly connected to computing device 105, and/or may beaccessed by computing device 105 over a network (not shown). In someembodiments, the computing device 105 and data store 110 may becollocated and accessed by the AR display 150 over a network.

Computing device 105 may include a processing device, memory, secondarystorage, one or more input devices (e.g., such as a keyboard, mouse,tablet, speakers, or the like), one or more output devices (e.g., adisplay, a printer, etc.), and/or other hardware components. Computingdevice 105 may be connected to data store 110 either directly or via anetwork. The network may be a local area network (LAN), a public widearea network (WAN) (e.g., the Internet), a private WAN (e.g., anintranet), or a combination thereof. The computing device 105 may beintegrated into the AR display 150 or image capture device 160 in someembodiments to improve mobility.

Data store 110 may be an internal data store, or an external data storethat is connected to computing device 105 directly or via a network.Examples of network data stores include a storage area network (SAN), anetwork attached storage (NAS), and a storage service provided by acloud computing service provider. Data store 110 may include a filesystem, a database, or other data storage arrangement.

The AR display 150 may include lenses through which a wearer (e.g., adental practitioner) may see a physical, real-world environment (e.g., apatient's oral cavity) and a projector for projecting visual elementsonto the lenses. Examples of AR displays include HoloLens®, GoogleGlass®, Vuzix Smart Glasses®, and Sony SmartEyeGlass®. The AR display150 may therefore overlay information for a dental practitioner onto thelenses in a position in the field of view of the practitioner thatcorresponds to a location of an identified area of interest. Todetermine where to display information, the AR display 150 may includeone or more sensors to track the eyes of a user and/or determine aposition of the user in relation to positions of objects viewed by theuser. The AR display 150 may also use images provided from image capturedevice 160 to determine where to display information to the dentalpractitioner. In some embodiments the image capture device 160 ismounted to the AR display 150.

As a dental practitioner wearing the AR display 150 views a patient,image capture device 160 may generate a stream of images that show thepatient from the dental practitioner's point of view. The image capturedevice may be or include a charge-coupled device (CCD) sensor and/or acomplementary metal-oxide semiconductor (CMOS) sensor. The image capturedevice 160 may provide images or video to the computing device 105 forprocessing. For example, the image capture device 160 may provide imagesto the computing device 105 that the computing device analyzes todetermine areas of interest on a dental arch or otherwise in an oralcavity viewed by a dental practitioner. The image capture device 160 mayalso provide images to the computing device 105 or AR display 150 thatare used to coordinate the position of elements of a visual overlay todisplay on AR display 150 so that the visual overlay is superimposedover the real-world environment viewed by the dental practitioner. Insome embodiments, the images captured by image capture device 160 may bestored in data store 110. For example, the image data 135 may be storedin data store 110 as a record of patient history or for computing device105 to use for analysis of the patient. The image capture device 160 maytransmit the discrete images or video to the computing device 105.Computing device 105 may store the image data 135 in data store 110.

In some embodiments, the image capture device 160 providestwo-dimensional data. In some embodiments, the image capture device 160may provide three-dimensional data or stereoscopic image data that maybe processed to produce three-dimensional data. For example, the imagecapture device 160 may have two cameras with a known separation andknown imaging angles that simultaneously capture image data. Thestereoscopic image data may be provided to computing device 105 as asingle stream of image data or as two separate streams of image data.The stereoscopic image data may be used to provide an estimation ofdepth for objects viewed through the AR display 150. For example, thecomputing device 105 may use the stereoscopic image data to identify athree dimensional location of a tooth in the field of view of the imagecapture device 160.

The image capture device 160 may include high definition cameras toaccurately capture the structure of areas of interest of a patient. Insome embodiments, the image capture device 160 may have one or morecameras that capture a wide field of view and additional cameras forcapturing a narrow field of view (e.g., for a region identified ascontaining an area of interest). In some embodiments, the image capturedevice 160 may include additional cameras to provide additional streamsof image data. Additional cameras may be used to improve threedimensional image quality.

In some embodiments, the image capture device 160 may include one ormore light sources to illuminate a patient for capturing images. Suchlight sources may include infrared, ultraviolet, or other wavelengthlight sources (e.g., LEDs or the like). These light sources mayilluminate an oral cavity to provide additional data over informationavailable from the visible light spectrum. For example, certainwavelengths such as infrared or ultraviolet wavelengths may more clearlyshow certain dental conditions such as plaque or cavities. In addition,in some embodiments, light sources may provide structured light toenhance three dimensional mapping of image data received from imagecapture device 160. For example, the light sources may project lines ora grid onto viewed objects to provide additional information about depthto the computing device 105.

The computing device 105 may include AR processing module 108. The ARprocessing module 108 may analyze image data 135 from a data store 110or directly from an image capture device 160. The AR processing module108 may then identify areas of interest to present in a visual overlayon AR display 150 and/or generate additional information to present onthe AR display 150. The information provided on an AR display 150 maydepend on a procedure to be performed, a wearer of the AR display 150,information known about a patient, and soon. For example, during aroutine checkup, the computing device 105 may provide patient history toa dental practitioner and/or display areas of interest identified basedon image data 135. In some embodiments, the dental practitioner mayinput the identity of a procedure to be performed into AR processingmodule 108. For this purpose, the dental practitioner may choose theprocedure from a number of preset options on a drop-down menu or thelike, from icons or via any other suitable graphical input interface, orby speaking commands to the AR system. Alternatively, the identity ofthe procedure may be input in any other suitable way, for example bymeans of preset code, notation or any other suitable manner, ARprocessing module 108 having been suitably programmed to recognize thechoice made by the user.

By way of non-limiting example, dental procedures may be broadly dividedinto prosthodontic (restorative) and orthodontic procedures, and thenfurther subdivided into specific forms of these procedures.Additionally, dental procedures may include identification and treatmentof gum disease, sleep apnea, and intraoral conditions. The termprosthodontic procedure refers, inter alia, to any procedure involvingthe oral cavity and directed to the design, manufacture or installationof a dental prosthesis at a dental site within the oral cavity, or areal or virtual model thereof, or directed to the design and preparationof the dental site to receive such a prosthesis. A prosthesis mayinclude any restoration such as implants, crowns, veneers, inlays,onlays, and bridges, for example, and any other artificial partial orcomplete denture. The term orthodontic procedure refers, inter alia, toany procedure involving the oral cavity and directed to the design,manufacture or installation of orthodontic elements at a dental sitewithin the oral cavity, or a real or virtual model thereof, or directedto the design and preparation of the dental site to receive suchorthodontic elements. These elements may be appliances including but notlimited to brackets and wires, retainers, clear aligners, or functionalappliances. Any of these orthodontic procedures and/or dental proceduresmay be facilitated by the AR system described herein.

In one embodiment, AR processing module 108 includes one or more area ofinterest (AOI) identifying modules 115, an AR display module 118, and atreatment control module 120. Alternatively, the operations of one ormore of the AOI identifying modules 115, AR display module 118, and/ortreatment control module 125 may be combined into a single module and/ordivided into multiple modules.

AOI identifying modules 115 are responsible for identifying areas ofinterest (AOIs) from image data 135 received from image capture device160. The image data may be images of a patient's oral cavity viewed by adental practitioner wearing the AR display 150. The AOI identifyingmodules 115 may also identify AOIs from reference data 138, which mayinclude patient history, virtual 3D models generated from intraoral scandata, or other patient data. Such areas of interest may include areasindicative of tooth wear, areas indicative of tooth decay, areasindicative of receding gums, a gum line, a patient bite, a margin line(e.g., margin line of one or more preparation teeth), and so forth.Areas of interest may also include areas indicative of foreign objects(e.g., studs, bridges, etc.), areas for the dental practitioner toperform planned treatment, or the like. The AOI identifying modules 115may, in identifying an AOI, analyze patient image data 135. The analysismay involve direct analysis (e.g., pixel-based and/or other point-basedanalysis), the application of machine learning, the application of imageregistration, and/or the application of image recognition. The AOIidentifying modules 115 may identify areas of interest directly from theimage data 135 received from the image capture device 160 or based on acomparison of the received image data 135 and reference data 138 orprevious patient data 140. For example, an AOI identifying module 115may use one or more algorithms or detection rules to analyze the shapeof a tooth, color of a tooth, position of a tooth, or othercharacteristics of a tooth to determine if there is any AOI that shouldbe highlighted for a dental practitioner.

AR display module 118 is responsible for determining how to presentand/or call out the identified areas of interest on the AR display 150.AR display module 118 may provide indications or indicators highlightingidentified AOIs. The AR display module 118 may determine a position toproject a virtual object in a visual overlay on an AR display 150 suchthat the overlay is positioned in the line of sight of the dentalpractitioner over the AOI. The virtual object may include text, numbers,a contour, colors, graphical images and/or other virtual objects. Forinstance, the AR display module 118 may determine from the position ofthe AOI in the image data 135 a corresponding position to project anindicator or indication on the AR display 150. As an example, the ARdisplay module 118 may provide an indication of wear on a tooth byhighlighting the worn area on the tooth in a notable color (e.g., thatcontrasts with a background on which the indication is superimposed)and/or or by providing an indicator pointing to the tooth. In someembodiments, the AR display 150 may provide additional indicatorsseparate from a position corresponding to the AOI in order to provideadditional data to a dental practitioner.

The AR display module 118 may provide the indications in the form offlags, markings, contours, text, images, and/or sounds (e.g., in theform of speech). In some embodiments, the AR display module 118 mayprovide a contour (e.g., via contour fitting) so as to follow a toothcontour or gingival contour in the image data 135. As an illustration, acontour corresponding to a tooth wear diagnostic assistance indicationmay be placed so as to follow a contour of the worn tooth. A contour mayalso follow a previous contour of the tooth or other dental feature. Forexample, a visual overlay may include a contour showing a previous shapeof a tooth, or a difference between a previous shape of a tooth and acurrent shape of the tooth. Such a contour may be placed in the visualoverlay so as to be superimposed over the real-world view of the toothin question or adjacent (e.g., touching) the tooth in question. As anillustration, a contour corresponding to a previous or future positionof a tooth may be displayed so as to follow the projected path of thetooth portion which is missing, or a contour corresponding to missinggingival scan data may be placed so as to follow the projected path ofthe gingival portion which is missing.

The wearer of the AR display 150 may provide an indication of anidentity of the wearer (e.g., through a menu or other user interface).AR processing module 108 may then determine what information to includein the visual overlay based on the identity of the wearer. For example,first information may be shown to a dentist and second information maybe shown to a dental hygienist. In some instances, the AR processingmodule 108 provides a script of actions for the dental practitioner ordental hygienist to perform and/or a script of things to say to thepatient. This script may have been input by a dentist, for example. Thescript may show up as a visual overlay on the AR display 150. The scriptmay be presented to the dental practitioner when particular eventsoccur, such as when a particular dental condition is identified fromimage data generated by image capture device 160. Additionally, oralternatively, the AR processing module 108 may walk a dentist or dentalhygienist through a patient history while the dentist views thepatient's mouth. An audio output describing the history may be output tothe dentist while one or more areas of interest associated with thedental history are highlighted to the dentist on the AR display 150 viathe visual overlay.

In some embodiments, a treatment control module 120 is responsible fordetermining what data to present on AR display 150 based on an intraoraltreatment or procedure of a patient. In some embodiments, the treatmentcontrol module 120 may also control one or more dental tools orinstruments that are used by a dental practitioner during treatment.This may include powering on the tools, powering off the tools, changingsettings of the tools, and so on. The treatment control module 120 mayaccess patient data 140, image data 135, and reference data 138 todetermine AR elements to provide on AR display 150. In some embodiments,the treatment control module 120 may receive AOIs from one or more AOIidentifying modules 115 or provide data or instructions to one or moreAOI identifying modules 115 to direct the AOI identifying modules 115 toidentify AOIs relevant to a particular treatment or part of a treatment.The treatment control module 120 may also provide tracking of dentaltools or other instruments in the view of image data 135 received fromimage capture device 160.

In one embodiment, the AR system 100 additionally includes a virtualreality (VR) display 152 that may be worn by a patient. The image datafrom the image capture device 160 and/or the visual overlay generatedbased on the image data may be output to the VR display 152. This mayenable the patient to view dental conditions of his teeth or gums that adental practitioner is seeing (and possibly describing). This mayfacilitate an explanation of the dental conditions to the patient by thedental practitioner. Image data from the image capture device and/orvisual overlays may also be sent to the VR display, for example, duringdental procedures.

In one embodiment, the patient is provided a control to select one ormore viewing modes for the VR display worn by the patient. One viewingmode shows the visual overlay generated by the AR processing module 108.This may include, for example, a virtual 3D model generated based on anintraoral scan while the intraoral scan is being performed. One viewingmode shows the view of the dental practitioner (e.g., the image datafrom the image capture device of the AR display worn by the dentalpractitioner) with the visual overlay generated by the AR processingmodule 108. One viewing mode shows the view of the dental practitionerwithout the visual overlay. One viewing mode shows entertainment contentfor the patient, such as movies.

In one embodiment, the AR system 100 includes an intraoral scanner 180.The computing device 105 may be a computing device connected to theintraoral scanner 180 that includes an intraoral scan application 109for controlling an intraoral scan procedure. The AR display 150 may bean AR display for the intraoral scanner 180.

In one embodiment, the intraoral scanner 180 includes an image sensor, acommunication module and one or more inputs (e.g., buttons, a touchsensor, switches, sliders, etc.). The image sensor generates intraoralimages of a patient and the communication module transmits thoseintraoral images to computing device 105. The computing device may thendisplay the intraoral images or a representation of the dental arch ofthe patient generated from the intraoral images (e.g., a virtual 3Dmodel of a dental site of the patient) via a visual overlay sent to theAR display 150. A user may then use the one or more inputs from theintraoral scanner, motion gestures, or other inputs to manipulate theintraoral images or the representation (e.g., virtual 3-D model)generated from the intraoral images. The intraoral images or virtual 3-Dmodel may be shown in the AR display as they are manipulated.

Intraoral scanner 180 may include a probe (e.g., a hand held probe) foroptically capturing three dimensional structures (e.g., by confocalfocusing of an array of light beams). Intraoral scanner 180 may alsoinclude other components such as optical components, an accelerometer,communication components, a gyroscope, processing devices, and so on.One example of an intraoral scanner 180 is the iTero® intraoral digitalscanner manufactured by Align Technology, Inc.

The intraoral scanner 180 may be used to perform an intraoral scan of apatient's oral cavity. Intraoral scan application 109 running oncomputing device 105 may communicate with intraoral scanner 180 toeffectuate the intraoral scan. A result of the intraoral scan may be asequence of intraoral images that have been discretely generated (e.g.,by pressing on a “generate image” button of the scanner for each image).Alternatively, a result of the intraoral scan may be one or more videosof the patient's oral cavity. An operator may start recording the videowith the intraoral scanner 180 at a first position in the oral cavity,move the intraoral scanner 180 within the oral cavity to a secondposition while the video is being taken, and then stop recording thevideo. The intraoral scanner 180 may transmit the discrete intraoralimages or intraoral video to the computing device 105. Computing device105 may store and/or process the discrete intraoral images or intraoralvideo in data store 110.

The manner in which the oral cavity of a patient is to be scanned maydepend on the procedure to be applied thereto. For example, if an upperor lower denture is to be created, then a full scan of the mandibular ormaxillary edentulous arches may be performed. In contrast, if a bridgeis to be created, then just a portion of a total arch may be scannedwhich includes an edentulous region, the neighboring abutment teeth andthe opposing arch and dentition. Thus, the dental practitioner may inputthe identity of a procedure to be performed into the intraoral scanapplication 109. For this purpose, the dental practitioner may choosethe procedure from a number of preset options on a drop-down menu or thelike that may be shown via the AR display. The dental practitioner maygenerate a treatment plan that includes one or more segments that are tobe scanned. A segment (or scan segment) may include a particular tooth(e.g., a preparation tooth), an upper or lower arch, a portion of anupper or lower arch, a bite, and so on.

The intraoral scan application 109 may provide a user interface that isshown in the AR display, where the user interface enables the dentalpractitioner to interact with intraoral scan application 109 throughmanipulation of graphical elements such as graphical icons and visualindicators such as buttons, menus, and so on while the dentalpractitioner remains focused on a patient (e.g., without looking awayfrom the patient to a computer monitor). Intraoral scan application 109may include a number of modes, such as a planning mode, a scan mode, animage processing mode, and a delivery mode. The intraoral scanapplication 109 may display different graphical elements via the ARdisplay 150 for each of the various modes.

Navigation or control of the user interface of the intraoral scanapplication 109 may be performed via user input. The user input may beperformed through various devices, such as a touch sensor on theintraoral scanner 180, gesture inputs detectable by the intraoralscanner 180, additional input mechanisms on the intraoral scanner 180,and so on. Navigation of the user interface may involve, for example,navigating between various modules or modes, navigating between varioussegments, controlling the viewing of the 3D rendering, or any other userinterface navigation.

Intraoral scan application 109 may include a planning mode that allows auser (e.g., dental practitioner) to generate a patient profile and/ortreatment plan for a patient. The patient profile may includeinformation such as patient name, patient contact information, patientdental history, and so on. The treatment plan may include dentalprocedures to be performed and/or teeth to which the dental proceduresare to be performed. Some treatment plans include an indication ofspecific patient teeth that are to be preparation teeth. Information forthe treatment plan may be shown in the AR display during the treatmentplanning mode.

Once a patient profile and/or treatment plan are generated, intraoralscan application 109 may enter a scan mode. A user may transition fromthe planning mode to the scan mode by navigating a menu displayed in theAR display 150. The scan mode allows the dental practitioner to captureimages and/or video (e.g., for lower arch segment, upper arch segment,bite segment, and/or preparation tooth segments). The images and/orvideo may be used to generate a virtual 3D model of a dental site. Whilein the scan mode, intraoral scan application 109 may register and stitchtogether intraoral images from the intraoral scanner 180 and generate apartial virtual 3-D model of a portion of a dental arch that has beenscanned thus far. Intraoral scan application 109 may interface with ARdisplay module 118 to cause AR display module 118 to then generate avirtual overlay that includes the partial virtual 3-D model of theportion of the dental arch. AR display module 118 may determine anappropriate region in a dental practitioner's field of view to projectthe partial virtual 3-D model, and may generate a virtual overlay withthe partial virtual 3-D model at the determined region. This virtualoverlay 118 may then be sent to AR display 150, and the dentalpractitioner 150 may see the progress of the intraoral scan during thescan.

During the scan mode, intraoral scan application 109 may provide thepartial virtual 3-D model to one or more of the AOI identifying modules115. The AOI identifying modules 115 may determine portions of thedental arch that have been scanned. The AOI identifying modules 115 maythen determine what areas in image data 135 received from the imagecapture device 160 associated with the AR display 150 correspond to thealready scanned portions of the dental arch. AR display module 118 maythen generate a virtual overlay the causes the already scanned portionsof the dental arch as viewed by the dental practitioner to behighlighted by superimposing colors over the scanned portions of thedental arch and/or that causes the not yet scanned portions to behighlighted. The visual overlay that is superimposed over portions ofthe patient's dental arch may be generated instead of or in addition tothe visual overlay that provides a virtual 3-D model of the scannedportions of the dental arch.

Once an intraoral scan is complete, intraoral scan application 109 mayenter an image processing mode. While in the image processing mode, theintraoral scan application 109 may process the intraoral scan data fromthe one or more scans of the various segments to generate a virtual 3Dmodel of a scanned dental site.

In one embodiment, intraoral scan application 109 performs imageregistration for each pair of adjacent or overlapping intraoral images(e.g., each successive frame of an intraoral video). Image registrationalgorithms are carried out to register two adjacent intraoral images,which essentially involves determination of the transformations whichalign one image with the other. Image registration may involveidentifying multiple points in each image (e.g., point clouds) of animage pair, surface fitting to the points of each image, and using localsearches around points to match points of the two adjacent images.Intraoral scan application 109 may repeat image registration for alladjacent image pairs of a sequence of intraoral images to obtain atransformation between each pair of images, to register each image withthe previous one. Intraoral scan application 109 then integrates allimages into a single virtual 3D model of the dental arch (or portion ofthe dental arch) by applying the appropriate determined transformationsto each of the images. Each transformation may include rotations aboutone to three axes and translations within one to three planes.

While in the image processing mode, a user may view the virtual 3D modelin detail to determine if it is acceptable. Intraoral scan application109 may invoke AR display module 118 to cause AR display module 118 togenerate a virtual overlay that includes the virtual 3D model, which maybe sent to the AR display 150. The image processing mode allows thedental practitioner to view the scans in detail at various angles byrotating, moving, zooming in or out, etc. of the virtual 3D model. Thedental practitioner may make a determination whether the quality of thescans are adequate, or whether particular segments or portions ofsegments should be rescanned. The dental practitioner may also navigateback to the scan mode to perform additional scans.

Once the scans are complete, a delivery mode allows the dentalpractitioner to send the scans and/or virtual 3D model out to anexternal facility to process the scans or 3D model.

FIG. 1B illustrates one embodiment of an augmented reality processingmodule 108, in accordance with an embodiment. The AR processing module108 may correspond to AR processing module 108 of FIG. 1A inembodiments. AR processing module 108 receives as an input image data162 from image capture device 160 associated with AR display 150,processes the image data 162, and generates a visual overlay 164 that isthen output to the AR display 150. The image data 162 preferablyincludes an image of a patient's oral cavity that includes a dental arch(or two dental arches). AR processing module processes the image data162 to determine areas of interest, where the areas of interest areareas in the oral cavity (e.g., on the dental arch) that potentiallyhave a clinical significance. For example, the AR processing module 108may identify possible cavities, tooth discoloration, gum discoloration,tooth cracks, tooth wear, gum recession, oral cancer, and so on. ARprocessing module 108 generates indicators for the identified areas ofinterest and adds those indicators to the visual overlay 164. Notably,the image data 162 may represent a real-world scene as viewed by adental practitioner wearing an AR display. AR processing module 108 mayreceive the image data 162, process the image data, and output thevisual overlay 164 to the AR display in real time or near-real time sothat the visual overlay corresponds to the scene that the dentalpractitioner is currently viewing through the AR display. The ARprocessing module 108 may receive a stream of image data 162 from theimage capture device 160 and may output a stream of the visual overlay164 that corresponds to the incoming stream of image data 162. Thus thevisual overlay 164 may be continually updated in real time or near-realtime to maintain correspondence to the scene as viewed by the dentalpractitioner as a patient moves, the dental practitioner moves, or thescene otherwise changes.

In one embodiment, AR processing module 108 includes multiple AOIidentifying modules 115. Alternatively, one or more of these AOIidentifying modules 115 may be combined into a single AOI identifyingmodule 115. Each AOI identifying module 115 is configured to identifyparticular types of information from the image data and/or particularAOIs for flagging from the image data. For example, AOI identifyingmodules 115 may identify an image of a tooth, dental arch, or otherdentition feature in the image data 162. In some embodiments, all of AOIidentifying modules 115 are implemented to identify AOIs using a varietyof techniques. In other embodiments, AR processing module 108 mayinclude a subset of the AOI identifying modules 115 using a subset ofthe AOI identifying techniques described herein. For example, the ARprocessing module 108 may include a dental arch/oral cavity identifier166, a dental arch segmenter 172 and one or more dental conditionidentifiers 174, but may lack a prior data comparator 180.

Dental arch/oral cavity identifier 166 may be responsible foridentifying an oral cavity in received image data 162 and foridentifying a dental arch in the oral cavity. To identify the oralcavity, dental arch/oral cavity identifier 166 performs image processingon the image data 162 using image recognition techniques. For example,oral cavities have visual cues that can be used to pinpoint the oralcavities in the image data 162. Dental arch/oral cavity identifier 166may include an oral cavity profile that may have been generated usingmachine learning techniques such as neural networks. Processing theimage data 162 may include first pre-processing the image data such asby performing re-sampling in a new coordinate system, performing noisereduction, enhancing contrast, adjusting scale, etc. Processing theimage data 162 may additionally include performing feature extraction toidentify lines, edges, ridges, point clouds, corners, point blobs, andso on. Processing the image data 162 may additionally include performingdetection and/or segmentation to select those lines, edges, ridges,point clouds, corners, point blobs, etc. that represent the oral cavityand/or objects within the oral cavity.

Dental arch/oral cavity identifier 166 can identify the dental arch (ormultiple dental arches) in the oral cavity using similar techniques asdescribed for identifying the oral cavity. However, a dental archprofile may be used to identify the dental arch.

In an example, dental arch/oral cavity identifier 166 may identifyfeatures in the image data based on geometric analysis of the image data162. The dental arch/oral cavity identifier 166 may perform geometricanalysis based on identification of lines or color blobs in the imagedata 162. The geometric analysis may identify the features of an oralcavity and/or the features of a dental arch.

Dental arch segmenter 172 may be responsible for segmenting anidentified dental arch into individual teeth. The dental arch segmenter172 may operate on similar principles as the dental arch/oral cavityidentifier. Dental arch segmenter 172 may receive a subset of image data162 that has already been processed by dental arch/oral cavityidentifier 166 (e.g., point blobs, contours, ridges, corners, pointclouds, etc. that represent a dental arch), and may perform detectionand segmentation to segment the dental arch into the individual teeth.Dental arch segmenter 172 and/or dental arch/oral cavity identifier 166may additionally identify gums in the oral cavity represented in theimage data 162 and separate the gums from the teeth.

The AOI identifying modules 115 additionally include one or more dentalcondition identifiers 174. Each dental condition identifier 174 may beresponsible for identifying a particular dental condition in the oralcavity of the patient from the image data 162 using one or more rules(that may include algorithms, models and/or profiles) that are tailoredto detection of that particular dental condition. Alternatively, asingle detection rule or set of rules (e.g., that may includealgorithms, models and/or profiles) may be used to detect multipledifferent types of dental conditions. The dental condition identifiers174 may operate on the original unprocessed image data 162 or mayoperate on processed image data that has been processed by the dentalarch/oral cavity identifier 166 and/or the dental arch segmenter 172.For example, one dental condition identifier 174 may be a broken toothidentifier, which may separately perform broken tooth identification foreach tooth identified by dental arch segmenter 172. Examples of dentalcondition identifiers 174 include a broken tooth identifier, a plaqueidentifier, a tooth wear identifier, an oral cancer identifier, a gumdiscoloration identifier, a tooth discoloration identifier, amalocclusion identifier, a gum recession identifier, a swollen gumidentifier, and so on.

In some embodiments, one or more dental condition identifier 174 useimage data generated after light having a specified wavelength is usedto illuminate the dental arch. The dental condition identifier 174 maysend an instruction to one or more light sources (not shown) that may bemounted to the AR display 150 or may be separate from the AR display150. The light sources may emit ultraviolet light, infrared radiation,or other wavelength radiation. The dental condition identifier 174 mayprocess the image data generated during such illumination of the dentalarch to determine additional information relevant to a particular dentalcondition. The specific wavelength may improve detection of theparticular dental condition.

In some embodiments, one or more of the dental condition identifiers 174performs a color analysis of the image data to identify a dentalcondition.

In some embodiments, the dental condition identifiers 174 may applyrules (e.g., including algorithms, models and/or profiles) that comparedentition features (also referred to as dental features) from a receivedimage to reference data 190, which may include a store of dentitionfeatures. The reference data 190 may include elements comprising modelsor images of dentition features. In some embodiments, the reference data190 may have other representations of dentition features. The elementsin reference data 190 may have AOIs associated with the dentitionfeatures. For example, a model of a tooth with a crack may be stored inthe reference data 190 with an associated indicator that the particulartooth had a crack. In some embodiments, the dental condition identifier174 may extract a model, image, set of edges, a point blob, set ofcontours, and/or other representation of a dentition feature from theimage data 162. The dental condition identifier 174 may then compare theextracted model, set of edges, point blob, set of contours, image orother representation of a dentition feature to a data store of similardentition features. A most similar stored dentition feature may beselected based on a point by point comparison, edge comparison, or othercomparison of the extracted feature representation to therepresentations of dentition features in the data store. The dentalcondition identifier 174 may then determine that the extracted dentitionfeature has the same AOIs present in the most similar stored dentitionfeature.

In some embodiments, the dental condition identifiers 174 may performanalysis of a dentition feature using machine learning algorithms. Forexample, a dental condition profile 192 may be trained based onreference data 190 to correlate dentition features in the reference data190 with associated clinical diagnosis of AOIs. The dental conditionidentifier 174 may then provide an image or an extracted representationof a dentition feature to the dental condition profile 192 and receivean indication of potential AOIs. In some embodiments, the dentalcondition identifier 174 may perform additional analysis to confirm theAOIs identified by a dental condition profile.

In some embodiments, dental condition identifiers 174 may use a dentalcondition profile 190 that has been trained using machine learningtechniques to identify a particular dental condition. A dental conditionprofile 192 may be trained by extracting contents from a training dataset and performing machine-learning analysis on the contents to generatea classification model and a feature set for the particular dentalcondition. Each dental condition profile may be or include a tailoredalgorithm for identifying a particular type of dental condition ormultiple different types of dental conditions. The training data setincludes positive examples of a dental condition (e.g., images in whichthe dental condition is present such as images of broken teeth) andnegative examples that lack the dental condition (e.g., images ofunbroken teeth). To generate the classification model and feature setfor a dental condition profile, the positive examples of the dentalcondition and the negative examples of the dental condition in thetraining data set are analyzed to determine the frequency of occurrenceof features (e.g., particular arrangements of point clouds, edges,contours, point blobs, etc.) in the positive examples and in thenegative examples. Positive features and negative features may then beranked based on, for example, frequency of occurrence in the positiveexamples and negative examples. These features make up a feature set forthe dental condition profile 192. The classification model for thedental condition profile 192 is generated based on the feature set andthe training data set. The classification model is a statistical modelfor data classification that includes a map of support vectors thatrepresent boundary features. The boundary features may be selected fromthe feature set, and may represent the highest ranked features in thefeature set.

A dental condition identifier 174 applies the received image data 162 orprocessed portion of the image data 162 (e.g., features such as pointblobs, contours, edges, corners, etc. that represent a tooth) to anappropriate dental condition profile 192. Based on the application ofthe image data 162 or features of the images data 162 to the dentalcondition profile 192, the dental condition identifier 174 classifiesthe image data or features as having the dental condition or not havingthe dental condition. The dental condition identifier 174 alsodetermines a contour of the dental condition in the image data 162. Thismay include identifying a contour of a crack, identifying a contour of aworn portion of a tooth, identifying a contour of a discolored gum area,identifying a contour of a possible oral cancer, and so on. The contourmay represent an intraoral area of interest to call to the attention ofa dental practitioner. In addition, the dental condition identifierdetermines a confidence level for the determined classification. If theconfidence value for the dental condition is 100%, then it is morelikely that the decision that the dental condition is present (or notpresent) is accurate than if the confidence value is 50%, for example.

One example dental condition identifier is a gum recession identifier.The gum recession identifier may detect gum recession by analyzing adistance between a patient's gum line and the crowns of one or moreteeth of a patient. The gum line may be determined by dental arch/oralcavity identifier 166 or the gum recession identifier. For example, thedental arch/oral cavity identifier 166 may use a dental arch profile toidentify and delineate a dental arch, including the gum line of thedental arch. Alternatively, the gum recession identifier may use a gumline profile that specifically identifies a contour of a gum line. Thetooth contours may be determined by the dental arch segmenter 172 or gumrecession identifier.

Once the gum line and the tooth contours are determined, a vector fromthe crown of a tooth to the gum line may be determined for one ormultiple teeth. The vector may be positioned at the center of the crownand may point in a direction towards the gum line that causes the vectorto approximately bisect the tooth. Vectors may be determined formultiple teeth that are visible in the received image data 162. Themagnitude of the vectors may be compared to an average gum recessionvalue. The average gum recession value may be based on a patient's age.If the magnitude of a vector exceeds the average gum recession value forthe patient's age, then the gum recession area (e.g., the gum line wherethe gum recession is identified) may be indicated as an area ofinterest. Additionally, the variance between the magnitude of the vectorand the gum recession value may be called out in the visual overlay.

In some embodiments, the AOI identifying modules 115 additionallyinclude a prior data comparator 180. The prior data comparator 180 mayidentify one or more areas of interest by comparing image data 162 toprior image data included in previous patient data 188. Patient data 188may include past data regarding the patient (e.g., medical records),previous or current scanned images or models of the patient, current orpast X-rays, 2D intraoral images, 3D intraoral images, virtual 2Dmodels, virtual 3D models, or the like.

Prior data comparator 180 may perform image registration between theimage data 162 and the prior image data of a patient's oral cavity,dental arch, individual teeth, or other intraoral regions. Imageregistration algorithms are carried out to register the current imagedata 162 from the image capture device of the AR system to one or moreprevious images of a patient's mouth, dental arch, teeth, etc. The imageregistration involves determination of the transformations which alignone image with the other. Image registration may involve identifyingmultiple points, point clouds, edges, corners, etc. in each image of animage pair, surface fitting to the points of each image, and using localsearches around points to match points of the two images. For example,prior data comparator 180 may match points of one image with the closestpoints interpolated on the surface of the other image, and iterativelyminimize the distance between matched points. Prior data comparator 180may also find the best match of curvature features at points of oneimage with curvature features at points interpolated on the surface ofthe other image, with or without iteration. Prior data comparator 180may also find the best match of spin-image point features at points ofone image with spin-image point features at points interpolated on thesurface of the other image, with or without iteration. Other techniquesthat may be used for image registration include those based ondetermining point-to-point correspondences using other features andminimization of point-to-surface distances, for example. Other imageregistration techniques may also be used.

Many image registration algorithms perform the fitting of a surface tothe points in adjacent images, which can be done in numerous ways.Parametric surfaces such as Bezier and B-Spline surfaces are common,although others may be used. A single surface patch may be fit to allpoints of an image, or alternatively, separate surface patches may befit to any number of a subset of points of the image. Separate surfacepatches may be fit to have common boundaries or they may be fit tooverlap. Surfaces or surface patches may be fit to interpolate multiplepoints by using a control-point net having the same number of points asa grid of points being fit, or the surface may approximate the points byusing a control-point net which has fewer number of control points thanthe grid of points being fit. Various matching techniques may also beemployed by the image registration algorithms.

In one embodiment, prior data comparator 180 may determine a point matchbetween images, which may take the form of a two dimensional (2D)curvature array. A local search for a matching point feature in acorresponding surface patch of another image is carried out by computingfeatures at points sampled in a region surrounding the parametricallysimilar point. Once corresponding point sets are determined betweensurface patches of the two images, determination of the transformationbetween the two sets of corresponding points in two coordinate framescan be solved. Essentially, an image registration algorithm may computea transformation between two images that will minimize the distancesbetween points on one surface, and the closest points to them found inthe interpolated region on the other image surface can be used as areference. The transformation may include rotations and/or translationalmovement in up to six degrees of freedom (e.g., rotations about one tothree axes and translations within one to three planes). Additionally,the transformation may include changes in image size (e.g., zooming inor out) for one or both of the images. A result of the imageregistration may be a transformation matrix that indicates therotations, translations and/or size changes that will cause the oneimage to correspond to the other image. In one embodiment, thetransformation matrix is applied to the prior image data to cause theprior image data to correlate with the current image data 162.

In some instances, the previous image data to which the current imagedata 162 is registered comprises a three dimensional model of apatient's dental arch and/or jaw. The three dimensional model may havebeen generated at a previous time based on an intraoral scan of thepatient's upper and/or lower dental arches. The three dimensional modelmay include the upper and lower dental arches, and may reflectarticulation of a patient's jaw and tooth contact points between theupper and lower dental arch. To register the image data 162 to the threedimensional model, prior data comparator 180 may digitally constructmultiple images of the three dimensional model from differentperspectives. If the image data is two-dimensional image data, then eachof the digitally constructed images may be two-dimensional images. Priordata comparator 180 may then attempt to register each of the digitallyconstructed images to the current image data 162 until registration issuccessful for one of the digitally constructed images. The perspectiveused to generate the registered digitally constructed image to the imagedata 162 is known, and so the three dimensional model may be registeredto the image data 162.

Once the prior image data has been registered to the current image data162 and transformed to match the current image data 162 as closely aspossible, the transformed previous image data (or a portion thereof) maybe used to generate visual overlay 164. Accordingly, a patient'shistorical dentition as represented in the previous image data may beadjusted to a current view point of a dental practitioner wearing an ARdisplay, and the visual overlay showing the patient's historicaldentition may be superimposed over the current view of the dentalpractitioner.

In one embodiment, once prior image data from previous patient data 188has been registered to the current image data 162 and transformedaccordingly, prior data comparator 180 compares the two images todetermine differences between the prior image data and the current imagedata 162. This may include performing any of the aforementioned imagerecognition techniques to identify features in the previous image dataand corresponding features in the current image data. For example, priordata comparator 180 may invoke dental arch/oral cavity identifier 166and/or dental arch segmenter 172 to identify a dental arch, individualteeth, a gum line, gums, etc. in the current image data 162 and previousimage data. Differences between the two images may be determined, andprior data comparator 180 may generate contours of those differences. Inone example, a difference between the two images for a tooth may includetooth wear reflected in the current image data 162 that is not shown inthe previous image data. In other examples, differences may include gumdiscoloration, tooth decay, tooth discoloration, gum recession, etc.that are shown in the current image data 162 but not in the previousimage data. Prior data comparator 180 may mark the contours of thedifferences between the images as areas of interest.

In one embodiment, the prior data comparator 180 identifies a feature inthe image data 162 to use to correlate the image data to the previousimage data. The feature may be a portion of a tooth, a gum line, aspecific tooth, or any feature of a dental arch. The prior datacomparator 180 may then compare the feature in the dental arch of theimage data 162 to the feature as represented in previous data associatedwith the dental arch. For example, if the feature is a tooth, the priordata comparator 180 may compare the tooth in the image data to the toothin previous patient data. The prior data comparator 180 may thendetermine if there has been excessive wear on the tooth, movement of thetooth, color change of the tooth, or other clinical determinations ofchange to the tooth. Additionally, prior data comparator 180 maydetermine whether an attachment was previously attached to a tooth butis no longer attached to the tooth (e.g., was lost). Additionally, priordata comparator 180 may determine whether an attachment has moved out ofposition (e.g., currently has a different position than it had wheninitially placed). If there has been a change, the prior data comparator180 may identify the change as an area of interest. In some embodiments,the prior data comparator 180 may also identify AOIs based on featurespreviously marked in patient history or by the dental practitioner.

In many instances, the prior image data will be stamped with a dateand/or time. Additionally, a current date and/or time may be determined.Prior data comparator 180 may determine a magnitude of a change in adental condition based on the determined differences between the currentimage data 162 and the previous image data. Additionally, prior datacomparator may model a rate or change of the dental condition based onthe magnitude of the difference. The accuracy of the modeled rate ofchange may be improved if the previous patient data includes previousimage data from multiple previous time periods.

Prior data comparator 180 may compare the determined magnitude of changeand/or the determined rate of change of the dental condition to generalnorms for the dental condition. The general norms may include rate ofchange thresholds for the dental condition. If the determined rate ofchange exceeds a rate of change threshold, then prior data comparator180 may generate a notice or flag for the dental practitioner callingout an abnormal change in the dental condition.

In one embodiment, AR processing module 108 includes a jaw modeldeterminer 181. Jaw model determiner 181 may be invoked to determine thearticulation of a patient's jaw. When jaw model determiner 181 isinvoked, it may notify a dental practitioner to instruct a patient tomove his or her lower jaw through multiple extremes of motion, to movehis or her face to multiple different positions, to smile, to open hisor her mouth, etc. This may include moving the lower jaw to the left asmuch as possible, moving the lower jaw to the right as much as possible,opening the mouth as far as possible, jutting the lower jaw forward asfar as possible (e.g., to cause an under bite), and positioning thelower jaw as far back as possible (e.g., to cause an over bite). Thismay also include moving the patient's head to the left to show a leftprofile view of the patient's face (with the mouth open and/or closed),moving the patient's head to the right to show a right profile view ofthe face (with the mouth open and/or closed), looking up, looking down,and so on.

The image capture device 160 may generate a stream of images while thepatient moves his or her jaw through the extremes of motion or positionextremes. Jaw model determiner 181 may determine from the stream ofimages those images that represent each of the motion or positionextremes. For example, jaw model determiner 181 may determine a leftprofile view with the patient's mouth closed, a left profile view withthe patient's mouth open, a right profile view with the patient's mouthclosed, a right profile view with the patient's mouth open, a front viewwith the patient's mouth closed, a front view with the patient's mouthopen (e.g., smiling), a view in which the mouth is opened as far aspossible, a view in which the lower jaw is moved to the right as much aspossible, a view in which the lower jaw is moved to the left as much aspossible, and so on.

Based on the jaw motion extremes and/or position extremes, jaw modeldeterminer 181 may generate an articulation model for the patient's jawthat defines motion vectors for the jaw. Alternatively or additionally,jaw model determiner 181 may generate another model of the jaw, such asa cephalographic model of the jaw and patient's head. The articulationmodel may be used along with a 3-D model of the patient's upper andlower arches to identify functioning contacts and interfering contactsbetween teeth in the upper arch and teeth in the lower arch. Thearticulation model may be used to track the movement of the jaw onvectors defined by the articulation model. Contacts between the teeth ofthe upper arch and the teeth of the lower arch may be determined fordifferent types of jaw motion such as shear movements, up and downmovements, etc. These contact points may be used to generate anocclusion map of the upper dental arch and lower dental arch.

In some instances, prior image data may include AOIs that have beenmarked in the prior image data by a dental practitioner. In suchinstances, the AOIs from the prior image data may be included in thevisual overlay at an appropriate location to call the dentalpractitioner's attention to the AOIs.

Prior data comparator 180 may additionally determine AOIs based on ananalysis of prior patient data other than prior image data of thepatient. For example, a clinical history for a patient might state thata particular procedure was performed on a specified tooth, that aspecified tooth in a concern, or provide other information about thepatient's dentition. Prior data comparator 180 may determine which toothin the image data 162 is referenced in the prior patient data, and maygenerate an indicator for an AOI corresponding to that tooth. In anexample, prior data comparator 180 may graphically walk the dentalpractitioner through a patient history, highlighting areas referenced inthe patient history using the visual overlay 164. Prior data comparator180 may also provide an audio output describing the patient history.

One type of patient data that the image data 162 may be compared to isan orthodontic treatment plan. The orthodontic treatment plan mayinclude a sequence of orthodontic treatment stages. Each orthodontictreatment stage may adjust the patient's dentition by a prescribedamount, and may be associated with a 3-D model of the patient's dentalarch that shows the patient's dentition at that treatment stage.Additionally, images of the patient's dentition may be generated at eachtreatment stage.

Prior data comparator 180 may compare the current image data 162 toprior images taken of the dental arch during previous orthodontictreatment stages. Additionally, or alternatively, the current image data162 may be compared to a 3-D model of the dental arch for the currenttreatment stage and/or one or more previous treatment stages of theorthodontic treatment plan. Based on the comparison of the current imagedata 162 to the 3-D model (or models) of the orthodontic treatment plan,progress of orthodontic treatment may be determined. Teeth movementprogress may then be shown virtually in the visual overlay 164 that isprovided to the AR display 150. Additionally, or alternatively, anindication of whether an action should be performed to continueorthodontic treatment may be determined, and the action may be suggestedvia the visual overlay 164.

Additionally, the current image data 162 may be compared to a finalorthodontic position (new arrangement of teeth) associated with a finaltreatment stage or other future orthodontic position associated withanother future intermediate treatment stage. Based on this comparison, avisual overlay showing the final arrangement or other future arrangementof the teeth may be generated. Accordingly, the dental practitioner maybe able to see what the patient will look like with his or her future orfinal tooth arrangement.

AR display module 118 is responsible for determining how to presentand/or call out the identified areas of interest on the AR display 150.AR display module 118 may provide indications or indicators highlightingidentified AOIs. In one embodiment, AR display module 118 includes avisual overlay generator 184 that is responsible for generating thevisual overlay 164 that is superimposed over a real-world scene viewedby a dental practitioner. The visual overlay generator 184 may determinea visual overlay for an AOI identified by one or more of the AOIidentifying modules 115, and may determine a position to project thevisual overlay 164 on an AR display 150 such that the visual overlay ispositioned in the line of sight of the dental practitioner over the AOIin the real-world scene viewed by the dental practitioner. For instance,the visual overlay generator 184 may determine from the position of theAOI in the image data 162 a corresponding position to project anindicator or indication on the AR display 150. As an example, the visualoverlay generator 184 may provide an indication of wear on a tooth byhighlighting the worn area on the tooth in a different color or byproviding an indicator pointing to the tooth. In some embodiments, theAR display module 118 may also provide additional indicators separatefrom a position corresponding to the AOI in order to provide additionaldata to a dental practitioner.

The AR display module 118 may provide the indications in the form offlags, markings, contours, text, images, and/or sounds (e.g., in theform of speech). In some embodiments, the AR display module 118 mayprovide a contour (e.g., via contour fitting) so as to follow a toothcontour or gingival contour in the image data 162. As an illustration, acontour corresponding to a tooth wear diagnostic assistance indicationmay be placed so as to follow a contour of the worn tooth.

In placing indications, AR display module 118 may or may not take intoaccount factors to avoid crowding the display. For instance the ARdisplay module 118 may display only one AOI on each portion of ARdisplay 150 or only a set number of AOI indicators on the AR display asa whole. In some embodiments, the AR display module 118 may take intoaccount available lighting, available angle, or other factorscorresponding to user viewing of the teeth and/or gingiva depiction, andmay position indicators to optimize the viewing for the dentalpractitioner. For example, the AR display module 118 may placeindicators such that they are not obstructing the view of the mouth orportion of a mouth of a patient.

The AR display module 118 may key the indications (e.g., via color,symbol, icon, size, text, and/or number). The keying of an indicationmay serve to convey information about that indication. The conveyedinformation may include classification of an AOI, a size of an AOIand/or an importance rank of an AOI. Accordingly, different flags orindicators may be used to identify different types of AOIs. For example,pink indicators may be used to indicate gingival recession and blueindicators may be used to indicate tooth wear. AR display module 118 maydetermine a classification, size and/or importance rank of an AOI, andmay then determine a color, symbol, icon, text, etc. for an indicator ofthat AOI based on the classification, size and/or importance rank.

Turning to keying which conveys indication size, the processing logicmay, in implementing such size-oriented keying, employ one or more sizethresholds. The origin of the thresholds may be set (e.g., by a dentalexpert) during a configuration operation and/or may be preset. In someimplementations, the thresholds may be set based on previous patientdata 188 or reference data 190. In some treatments, larger size of anAOI may be indicative of greater clinical importance. For example, alarge crack in a tooth may be worse than a small crack in a tooth.Similarly, a large amount of tooth wear or gum recession may be morecritical than a small amount of tooth wear of gum recession.Furthermore, a small area of wear may be less important than a largearea of wear. As an illustration, three thresholds might be set withrespect to a type of AOI. Implementation may be such that indicationsfalling into the largest of the three size thresholds are keyed redand/or with the numeral “1,” that indications falling into the smallestof the three size thresholds are keyed purple and/or with the numeral“3,” and/or that indications falling into the middle-sized of the threethresholds are keyed yellow and/or with the numeral “2.”

Turning to keying which conveys AOI classification, indicators mayidentify classifications assigned to intraoral areas of interest. Forexamples, AOIs may be classified as tooth wear, tooth cracks, toothpositions, gum recession, gingivitis, plaque, or other types of AOI.AOIs representing changes in patient dentition may represent toothdecay, receding gums, tooth wear, a broken tooth, gum disease, gumcolor, moles, lesions, tooth shade, tooth color, an improvement inorthodontic alignment, degradation in orthodontic alignment, and so on.Different criteria may be used for identifying each such class of AOI.For example, a change in orthodontic alignment may be identified basedon a planned orthodontic treatment, while tooth wear may be identifiedby an unnatural shape of a tooth.

In some embodiments, AR display module 118 includes a light enhancementmodule 182 that can improve the visibility of regions of an image withinan oral cavity. The light enhancement module 182 can then generate lightenhancement effects that improve the visibility of a real-world oralcavity viewed by a dental practitioner. These light enhancement effectscan be particularly beneficial during low light conditions.

Light enhancement module 182 may receive an indication of the oralcavity in the image data 162 from dental arch/oral cavity identifier166. Light enhancement module 182 may then apply a darkening effect toall regions outside of the oral cavity. For example, light enhancementmodule 182 may add a visual overlay 164 that includes dark pixels withtransparency. That way, regions outside of the oral cavity will appeardark to a dental practitioner when viewed through the AR display.Additionally, or alternatively, light enhancement module may apply lightenhancing effects to the region inside of the identified oral cavity. Bybrightening the regions in the oral cavity and/or darkening the regionsoutside of the oral cavity, a dental practitioner's view of the oralcavity may be improved. The human eye automatically dilates or contactsthe pupil to adjust for a total amount of light entering the eye.Additionally, the human eye automatically performs a white balance basedon the environment viewed by an eye. Light enhancement module 182 takesadvantage of these phenomena by adjusting the total amount of light andthe total colors that a dental practitioner wearing an AR display sees.Thus, even without actually adding more light to a patient's mouth, thelight enhancement module 182 may improve a dental practitioner's abilityto view the oral cavity by reducing the overall light seen by the eyeand increasing the relative amount of light that is caused by regionsinside of the oral cavity vs. regions outside of the oral cavity.

In one embodiment, the light enhancement module 182 determines whether adental practitioner wearing the AR display is focused on (e.g., lookingat) the patient's oral cavity. The AR display may include an additionalimage capture device to track the dental practitioner's eye movement,and the determination of whether the dental practitioner is focused onthe patient's oral cavity may be based on a direction of the eyes inrelation to a location of the oral cavity in the image data 162.Alternatively, the light enhancement module 182 may determine that thedental practitioner is focused on the oral cavity if an oral cavity isidentified to be near a center of the image. Alternatively, the lightenhancement module 182 may determine that the dental practitioner isfocused on the oral cavity if an oral cavity is detected anywhere in theimage data 162. Light enhancement module 182 may apply the lightenhancement effects while the dental practitioner is focused on thepatient's dental cavity. The light enhancement effects may thendisappear once the dental practitioner stops focusing on the patient'soral cavity (e.g., looks away from the patient).

In some embodiments, a light source may be placed inside of a patient'soral cavity. The light source may be a light emitting diode (LED), anincandescent light, a halogen light, or other type of light. The lightsource may help to illuminate the patient's oral cavity during a dentalprocedure. However, the light source may cause glare that may blind thedental practitioner. Accordingly, in embodiments light enhancementmodule 182 may generate a visual overlay 164 that blocks a brightness ofthe light source. For example, the light source may emit light at aparticular wavelength or wavelengths. The light enhancement module maygenerate a visual overlay 164 that acts as a filter to at leastpartially filter out particular wavelengths of light. Thus, the lightsource may illuminate the patient's oral cavity without blinding, orcausing glare for, the dental practitioner.

In some embodiments, AR processing module 108 provides a visual overlay164 during a dental procedure that facilitates that dental procedure.Any type of dental procedure can be facilitated by AR processing module108. Examples of dental procedures that might be facilitated includegrinding of an interproximal area between teeth (e.g., to make room fororthodontic treatment), grinding of a tooth into a stump (e.g., toenable a cap to be placed over the tooth), grinding of a crown of atooth (e.g., to improve contact points between an upper and lower dentalarch), an intraoral scan, drilling of a tooth (e.g., for a root canal orto place an implant anchor), and so on. Many other types of proceduresfor dentistry and orthodontics may also be improved by the AR processingmodule.

In some embodiments, a treatment control module 120 is responsible fordetermining what data to present on AR display 150 based on a treatmentof a patient (e.g., a dental procedure performed on the patient). Insome embodiments, the treatment control module 120 may also control oneor more dental tools or dental instruments that are used by a dentalpractitioner during treatment. The treatment control module 120 mayaccess previous patient data 188, image data 162, and/or reference data190 to determine AR elements to provide on AR display 150. In someembodiments, the treatment control module 120 may receive AOIs from AOIidentifying modules 115 or provide data or instructions to AOIidentifying modules 115 to direct the AOI identifying modules 115 toidentify AOIs relevant to a particular treatment or part of a treatment.The treatment control module 120 may also provide tracking of dentaltools or instruments in the image data 162 received from image capturedevice 160.

In some embodiments, a treatment control module 120 may determine atreatment or procedure that is being performed by a dental practitioner.The treatment or procedure may be identified based on input from thepractitioner such as in a user interface of the AR display 150 oranother user interface. In some embodiments, the treatment or proceduremay be identified from patient data 140 indicating a reason for acurrent appointment with the dental practitioner. The treatment orprocedure may also be selected by a dental practitioner based onrecommendations or indications provided on the AR display 150 by ARdisplay module 118 during an examination or treatment. The treatmentcontrol module 120 may access reference data 190 to determine particularAOIs to flag during the identified treatment or procedure and/or othergraphics (e.g., simulated objects) to display. For example, if thetreatment control module 120 determines that an implant is to beinserted onto the patient's dental arch, the treatment control module120 may determine steps of the procedure and simulated objects toprovide at different steps of a dental implant procedure. As anillustration, the treatment control module 120 may determine that theprocedure includes a drilling step and that an indication of a targetdrill direction and depth is to be displayed.

The treatment or procedure may also be identified based on specificapplications executing on the computing device, such as intraoral scanapplication 109. For example, the intraoral scan application 109 mayinterface with the AR processing module 108 during an intraoral scanprocedure. Based on this interaction, AOI identifying modules 115 maydetermine areas of interest associated with scanned portions and/orunscanned portions of a dental arch. Additionally, treatment controlmodule 120 may provide feedback for facilitating control of theintraoral scan procedure. Additionally, AR display module 118 maygenerate appropriate visual overlays to output to an AR display duringthe intraoral scan procedure.

During treatment, the treatment control module 120 may identify an AOIor receive an indication of an AOI from AOI identifying modules 115. Forexample, the AOI identifying modules 115 may provide image data, amodel, a contour of an AOI, or other representation of an AOI totreatment control module 120. In some embodiments, the treatment controlmodule 120 may identify AOIs for a particular treatment or procedurebased on image analysis of image data 162. The treatment control module120 may also instruct one or more of the AOI identifying modules 115 toidentify a particular AOI or type of AOI based on the treatment orprocedure. As the dental practitioner performs a treatment or procedure,the treatment control module 120 may receive updated image data, models,or representations of the AOI. The treatment control module 120 may thenidentify a change to the AOI based on the new data. For example, if theprocedure is for an interproximal reduction, the treatment controlmodule may identify a reduction in the size or shape of a tooth beingground based on the new data. The treatment control module 120 may theninstruct the AR display module 118 to generate an updated visual overlay164 for the AR display based on the change. For example, in aninterproximal reduction procedure, an area of a tooth to be ground maybe highlighted (e.g., superimposed over the tooth in the AR display)using a first color, and a new color may be superimposed onto the ARdisplay 150 when a threshold amount of material has been removed fromthe tooth to indicate to the dental practitioner that the appropriateamount of material has been removed.

In some embodiments, the treatment control module 120 may also controlone or more tools or instruments used by a dental practitioner. Forexample, during a drilling procedure, the treatment control module 120may lock use of the drill (or turn off the drill) if the drill is not inthe right position or has already drilled to a planned or recommendeddepth. The treatment control module 120 may additionally oralternatively power on or off the drill (or other dental tool) based ona position and/or orientation of the drill (or other dental tool). Thetreatment control module 120 may additionally or alternatively controlother settings of a drill or other dental tool based on the positionand/or orientation of the drill or other dental tool as determined fromthe image data 162. For example, an intensity setting for a laser drillmay be adjusted based on the position and/or orientation of the laserdrill. Similarly, in an interproximal reduction procedure, the treatmentcontrol module 120 may cause a grinding tool to stop grinding if aplanned amount of material has been removed.

In one embodiment, treatment control module 120 includes a toolidentifier/controller 170 and/or one or more dental procedurefacilitators 176. Each dental procedure facilitator 176 may beresponsible for assisting a particular type of dental procedure orintraoral treatment. For example, the dental procedure facilitators 176may include an intraoral scan facilitator that facilitates intraoralscanning, a dental drilling facilitator that facilitates dentaldrilling, a tooth grinding facilitator that facilitates tooth grinding(e.g., for interproximal reduction, generating a preparation tooth,improving a patient bite, etc.), an orthodontic treatment facilitatorfor facilitating one or more aspects of an orthodontic treatment, adental attachment facilitator, an implant insertion facilitator, and soon.

Each dental procedure facilitator 176 may perform different operationsbased on the dental procedure to be performed. Details about aparticular dental procedure for a particular patient may be included ina treatment plan 186 and/or previous patient data 188. The treatmentplan 186 may indicate the procedure to be performed, the tooth or otherarea on which the treatment is to be performed, and/or one or more otherparameters of the dental procedure. The dental procedure facilitator 176may process the treatment plan 186 and the image data 162 to determinegraphics to embed in the visual overlay 164 for facilitating the dentaloperation. Such graphics may include graphics indicating tooth materialto be removed, an area to be drilled, a location on a tooth to place anattachment, and so on. Such graphics may additionally or alternativelyindicate an ideal position and/or orientation for a dental tool toperform a dental procedure such as drilling or grinding. Graphicsshowing the ideal position/orientation for a dental tool may be used bymoving the dental tool until the dental tool as seen by the dentalpractitioner through the AR display is collocated with the graphicalindication for the position and orientation in the AR display. Graphicsmay additionally or alternatively indicate an insertion path for animplant, cap or bridge. These graphics may be added to the visualoverlay 164 and superimposed over a view of an oral cavity as seen by adental practitioner through an AR display. Some example dentalprocedures and the operations performed to facilitate those dentalprocedures using augmented reality are described below with reference toFIGS. 20-22.

One example dental procedure facilitator 176 is an insertionfacilitator. The insertion facilitator may perform operations tofacilitate insertion of a crown or bridge onto a dental arch. Theinsertion facilitator may compute an optimal insertion path for thecrown or bridge based on the geometries of surrounding teeth. This mayinclude determining an angle and direction of the insertion path.Alternatively, the insertion path may be specified in a treatment planthat has already been generated. Based on the insertion path, theinsertion facilitator determines a shape for one or more prep tooth onwhich the crown or bridge will be placed along with the insertion path.Insertion facilitator then marks or highlights the portions of the toothor teeth to be ground to create the prep tooth that will have thedetermined shape. The insertion facilitator may mark the reduction spacefor a crown or bridge in both the occlusion direction and an adjacent(proximal) direction. Once the one or more prep teeth are created, theinsertion path may be shown in the visual overlay.

One example dental procedure facilitator is an undercut reductionfacilitator. The undercut reduction facilitator may determine teeth withundercuts. The undercut facilitator may then highlight the portions ofthose teeth with undercuts that are causing the undercuts. Thesehighlights may be shown as a colored visual overlay in the shape of theundercut regions that is shown on the AR display. A dental practitionermay then grind down the highlighted area of the teeth the remove theundercuts.

One example dental procedure facilitator is a cavity drillingfacilitator. The cavity drilling facilitator may show an outline and/orhighlight of a cavity to be drilled, and indicate whether all of thecavity has been removed (e.g., by marking cavity areas with a firstcolor and healthy areas with a second color). The cavity drillingfacilitator may also show drill positions and orientations to remove thecavity.

One example dental procedure facilitator is an occlusion treatmentfacilitator. Occlusion refers to the contact between teeth. Moreparticularly, occlusion is the relationship between the maxillary(upper) teeth and the mandibular (lower) teeth, such as during chewing.A malocclusion is the misalignment of the teeth and/or jaw that impairsa person's bite. Contacts between maxillary teeth and mandibular teethmay be divided into functional contacts and interfering contacts. Anocclusion map as generated by jaw model determiner 181 may showfunctional contacts using a first color and interfering contacts using asecond color. This occlusion map may be registered to the dental archshown in the image data 162. Thus, functional contacts and interferingcontacts may each be highlighted in a visual overlay shown on the ARdisplay 150. A dental practitioner may grind down teeth at theinterfering contacts to improve the patient's occlusion (e.g., toeliminate a malocclusion). As the tooth is ground, new image data may bereceived in real time or near real time from the image capture device.The new image data may be used to compute a new occlusion map, and anupdated visual overlay showing the new occlusion map may be projectedonto the AR display 150 to overlay the new occlusion map over the dentalpractitioner's view of the dental arch.

One example dental procedure facilitator is an interproximal reduction(IPR) facilitator. The IPR facilitator may mark the portions of teeth inthe interproximal area between the teeth that is to be removed via thevisual overlay. As the teeth are ground, new image data may be receivedin real time or near-real time, and the markings showing theinterproximal area to be removed may be updated based on the new imagedata. The new markings may be included in an updated visual overlay thatis determined and sent to the AR display for display in real time ornear-real time. Additionally, the IPR facilitator may calculate an archlength and indicate if a planned amount of interproximal reeducation hasbeen achieved or cannot be achieved. The IPR facilitator may suggestadditional actions to take to generate a desired interproximal gap. Thissuggestion may be provided via the visual overlay. For example, the IPRfacilitator may determine an additional interproximal reduction toimplement between two additional teeth to generate additionalinterproximal space.

One example dental procedure facilitator is an implant facilitator. Theimplant facilitator may virtually mark an area to place a hole for animplant in the visual overlay 164. The area to place the implant may bepre-planned or may be computed by the implant facilitator. For example,the implant facilitator may determine a mid-distance between teeth, maydetermine an arch of one or more teeth, may determine a middle of thejaw, may determine the position of a pre-planned location, etc. Thesedeterminations may be used to compute where to place the hole for theimplant. If there is a CT scan of the jaw, then the bone portion of thedental arch that is not generally visible may be shown by registeringthe CT scan with the image data 162 and then including the CT scan datain the visual overlay 164. When the hole is to be drilled, the implantfacilitator may provide information to orient and place a drillproperly. As the drilling is performed, the implant facilitator mayindicate when a desired depth has been reached for the hole and/or maypower off the drill.

The tool identifier/controller 170 may include tool profiles that areusable to identify particular dental tools in the image data 162. Basedon the tool profiles, tool identifier/controller 170 may determine atype of tool, a position of the tool and an orientation of the tool. Adental procedure facilitator 176 may identify a dental procedure to beperformed, and may provide a visual indication of a position and/ororientation for a dental tool and/or for a position and/or shape ofmaterial to be removed from one or more teeth, material to be added tothe one or more teeth, an implant to be inserted, a hole to be drilled,and so on. Tool identifier/controller 170 may control a power and/or oneor more settings of a dental tool based on the position and/ororientation of the dental tool in relation to the dental procedure to beperformed.

In one embodiment, treatment control module 120 includes a hapticsmodule 177. A dental practitioner may wear haptics gloves that arecapable of providing forces, vibrations and/or motions to the dentalpractitioner. Alternatively, or additionally, dental tools used by thedental practitioner may include haptics components that can provide suchforces, vibrations and/or motions. AOI identifying modules 115 maygenerate haptic feedback areas based on the identified AOIs. The hapticfeedback areas may correspond to the locations of the AOIs and/or may benear the AOIs.

When tool identifier/controller 170 determines that a dental tool hasreached a haptic feedback area, haptics module 177 may send a signal tothe haptic gloves and/or haptics enabled dental tool to cause a hapticfeedback. The haptic gloves and/or dental tool may then provide a force,vibration or motion to indicate that the dental tool has reached thehaptic feedback area. For example, if a particular position andorientation for a dental tool is required for a dental procedure, thenthe haptics module 177 may cause a haptic feedback to be provided whenthe dental tool achieves that position and/or orientation. Toolidentifier/controller 170 may determine when the dental tool has reachedthat target position and orientation based on analysis of the image data162. Similarly, haptic feedback may be provided during a drillingoperation when a target depth has been reached. Similarly, a hapticfeedback may be provided during one or more grinding operations when adesired amount of tooth removal at a particular area has been achieved.

In some embodiments, dental tools include sensors that may be used tofacilitate tracking of the dental tools. This may increase an accuracyof tracking the dental tools verses relying solely on tracking of thedental tools from the image data 162. For example, dental tools mayinclude accelerometers, gyroscopes, magnetic tracking sensors, imagecapture devices (e.g., complementary metal-oxide semiconductor (CMOS)sensors and/or charge-coupled device (CCD) image sensors). In theinstance of image capture devices on dental tools, images generated bysuch image capture devices may be generated in real time or near-realtime and registered against a 3-D model of a dental arch being operatedon. A position of the image capture device on the dental tool, a fieldof view of the image capture device, etc. may be known, and a relativeposition of the image capture device to a tool tip (e.g., head of agrinder or head of a drill bit) may be known. Accordingly, capturedimages may be registered to the 3-D model to accurately determine aposition and orientation of the dental tool relative to the dental arch.

In some embodiments, an intraoral scanner is used as an additionalsource of image data 163 during an intraoral procedure. The intraoralscanner may be positioned in the patient's oral cavity and pointedtoward an area on the dental arch where a dental procedure is beingperformed. For example, the intraoral scanner may be positioned so as totake images of a tooth that is being drilled or ground. The intraoralscanner may provide a high resolution image of the dental procedure froman angle or view that a dental practitioner would otherwise not haveaccess to. The image data from the intraoral scanner 163 may be receivedby AR processing module. AOI identifying modules 108 may then identifyareas of interest from the image data 163 in addition to identifyingareas of interest in image data 162. Additionally, toolidentifier/controller 170 may identify a dental tool from the image data163 and/or determine additional information about the dental tool fromthe image data 163 than can be determined from image data 162. ARdisplay module 118 may generate a zoomed in view of the dental procedurebased on the image data 163 received from the intraoral scanner. ARdisplay module 118 may determine a region in a dental practitioner'sview (e.g., a region of image data 162) that is outside of the oralcavity and dental arch. AR display module 118 may then generate a visualoverlay 164 that includes the zoomed in view of the dental procedurefrom the image data 163. The visual overlay 164 may place the zoomed inview on the AR display at the region of the dental practitioner's fieldof view that is outside of the oral cavity and dental arch. Accordingly,the dental practitioner may alternate between focusing on his real-worldphysical view of the patient's oral cavity and the zoomed in view of thedental procedure as appropriate during the dental procedure to improvehis or her accuracy at performing the dental procedure.

FIGS. 2-29 below describe example applications of AR enhancements for adental practitioner. The examples are described with reference to imagesrepresenting the AR display provided to a dental practitioner and/orflow charts describing processes of generating or providing such ARdisplays. In the examples, particular colors or types of indicators maybe used to highlight AOIs or other elements on an AR display. However,the particular colors or types of indicators are examples and othertypes or color indicators may be used according to various embodimentsdescribed herein. In addition, the flow charts provide example processesthat may be performed by an AR system. However, the processes performedby the AR system may include fewer or additional blocks than shown, andin some embodiments the processes in the flow charts may be performed ina different order than shown.

The methods depicted in FIGS. 2-29 may be performed by a processinglogic that may comprise hardware (e.g., circuitry, dedicated logic,programmable logic, microcode, etc.), software (e.g., instructions runon a processing device to perform hardware simulation), or a combinationthereof. Various embodiments may be performed by an AR system 100, acomputing device 105 and/or an AR processing module 108 as describedwith reference to FIGS. 1A-1B.

FIG. 2 illustrates a flow diagram for a method 200 of determining areasof interest by an augmented reality device based on comparison toprevious image data, in accordance with an embodiment. At block 210 ofmethod 200, processing logic receives image data of a dental arch froman image capture device of an augmented reality system. At block 215,processing logic identifies previous image data associated with thedental arch. The previous image data may be stored in a patient historydata store, for example. At block 220, processing logic registers theimage of the dental arch to previous image data associated with thedental arch. Image registration may be performed as described hereinabove.

At block 230, processing logic compares one or more areas of the dentalarch from the image to one or more corresponding areas of the dentalarch from the previous image data. In one embodiment, based on thecomparison processing logic determines a difference between an area ofthe dental arch in the image and a corresponding area of the dental archin the previous image data. For example, differences such as changes intooth wear, changes in gum recession, changes in gum color, changes intooth color, changes in gum swelling, and so on may be identified. Inone embodiment, processing logic may identify those changes that areover a threshold value for an amount of change.

At block 235, processing logic determines a position of an area ofinterest on the dental arch based on the comparison. The area ofinterest may be an area of the identified differences or an area of atooth or gum for which the difference was identified. At block 240,processing logic generates a visual overlay comprising an indication ofthe area of interest. In one embodiment, the indication of the area ofinterest may use a color scheme or other indicator to indicate themagnitude of change to the identified area of interest. At block 250,processing logic outputs the visual overlay to display of the augmentedreality system. The AOI in the visual overlay is superimposed on thedisplay over a view of the dental arch at the position of the area ofinterest.

FIG. 3 illustrates a flow diagram for a method 300 of registering imagedata from an image capture device of augmented reality device to a threedimensional model, in accordance with an embodiment. Method 300 may beperformed at block 220 of method 200. At block 310 of method 300,processing logic determines that previous image data comprises athree-dimensional (3-D) model of a dental arch. At block 320, processinglogic generates a plurality of perspective view images of the 3-D model.At block 330, processing logic compares the image of the dental arch tothe plurality of perspective view images to identify a perspective viewimage for which the model in the perspective view image most closelymatches the dental arch in the image. The image data may then beregistered to the 3-D model using the identified perspective view image.

FIG. 4 illustrates a flow diagram for a method 400 of determiningdifferences between a dental arch as depicted in image data from animage capture device of an augmented reality device and the dental archas depicted in previous image data, in accordance with an embodiment. Atblock 405 of method 400, processing logic compares one or more areas ofthe dental arch from the image to one or more corresponding areas of thedental arch from previous image data. Based on the comparison,processing logic may identify a difference between the received imagedata and the previous image data at one or more locations on the dentalarch. The difference may show a change in a dental condition such as anamount of tooth wear, an amount of gum recession, and so on. At block410 of method 400, processing logic determines a magnitude of change ofa condition from a tooth or gum between the previous image data and thecurrent image data.

At block 415, processing logic determines a previous date associatedwith the previous image data and a current date associated with currentimage data. At block 420, processing logic computes an amount of timebetween the previous date and the current date. At block 425, processinglogic computes a rate of change of a dental condition for the tooth orgum. At block 430, processing logic determines a target rate of changeof the condition for the tooth or gum. At block 435, processing logicdetermines that a difference between the target rate of change and therate of change that was identified exceeds a rate of change threshold.The rate of change threshold may be determined based on a healthy rateof change of the dental condition as viewed in a statisticallysignificant sample of patients. At block 440, processing logic generatesa visual overlay including the change in the dental condition and/or anindication that the difference between the target rate of change and therate of change that was identified exceeds the rate of change threshold.The visual overlay may be projected onto the AR display so that thechange is superimposed over an appropriate location of the tooth or gumthat has undergone the change as viewed by the dental practitioner. Thevisual overlay may be updated in real time or near-real time as updatedimage data is received from the image capture device.

FIG. 5 illustrates a flow diagram for a method 500 of tracking progressof an orthodontic treatment plan using image data from an augmentedreality device, in accordance with an embodiment. An orthodontictreatment plan may be generated based on an intraoral scan of a dentalarch to be modeled. The intraoral scan of the patient's dental arch maybe performed to generate a three dimensional (3D) virtual model of thepatient's dental arch. For example, a full scan of the mandibular and/ormaxillary arches of a patient may be performed to generate 3D virtualmodels thereof. The intraoral scan may be performed by creating multipleoverlapping intraoral images from different scanning stations and thenstitching together the intraoral images to provide a composite 3Dvirtual model. In other applications, virtual 3D models may also begenerated based on scans of an object to be modeled or based on use ofcomputer aided drafting techniques (e.g., to design the virtual 3Dmold). Alternatively, an initial negative mold may be generated from anactual object to be modeled. The negative mold may then be scanned todetermine a shape of a positive mold that will be produced.

Once the virtual 3D model of the patient's dental arch is generated, adental practitioner may determine a desired treatment outcome, whichincludes final positions and orientations for the patient's teeth.Processing logic may then determine a number of treatment stages tocause the teeth to progress from starting positions and orientations tothe target final positions and orientations. The shape of the finalvirtual 3D model and each intermediate virtual 3D model may bedetermined by computing the progression of tooth movement throughoutorthodontic treatment from initial tooth placement and orientation tofinal corrected tooth placement and orientation. For each treatmentstage, a separate virtual 3D model of the patient's dental arch at thattreatment stage may be generated. The shape of each virtual 3D modelwill be different. The original virtual 3D model, the final virtual 3Dmodel and each intermediate virtual 3D model is unique and customized tothe patient.

Accordingly, multiple different virtual 3D models may be generated for asingle patient. A first virtual 3D model may be a unique model of apatient's dental arch and/or teeth as they initially exist prior totreatment, and a final virtual 3D model may be a model of the patient'sdental arch and/or teeth after correction of one or more teeth and/or ajaw. Multiple intermediate virtual 3D models may be modeled, each ofwhich may be incrementally different from previous virtual 3D models.

Each virtual 3D model of a patient's dental arch may be used to generatea unique customized mold of the dental arch at a particular stage oftreatment. The shape of the mold may be at least in part based on theshape of the virtual 3D model for that treatment stage. Aligners may beformed from each mold to provide forces to move the patient's teeth. Theshape of each aligner is unique and customized for a particular patientand a particular treatment stage. In an example, the aligners can bepressure formed or thermoformed over the molds. Each mold may be used tofabricate an aligner that will apply forces to the patient's teeth at aparticular stage of the orthodontic treatment. The aligners each haveteeth-receiving cavities that receive and resiliently reposition theteeth in accordance with a particular treatment stage.

At block 510 of method 500, processing logic receives image data of adental arch from an image capture device of an augmented reality device.At block 515, processing logic determines a current treatment stage ofthe orthodontic treatment plan. At block 520, processing logic registersthe image of the dental arch to previous image data associated with thecurrent treatment stage. The previous image data associated with thecurrent treatment stage may include the three-dimensional virtual modelof the dental arch for the current treatment stage. Registration of thecurrent image data to the virtual 3-D model of the dental arch may beperformed as set forth in method 300.

At block 525, processing logic compares one or more areas of the dentalarch from the received image to one or more corresponding areas of thedental arch from the previous image data (e.g., from the virtualthree-dimensional model associated with the current treatment stage). Atblock 530, processing logic determines that a tooth deviates from theorthodontic treatment plan for the current treatment stage. Eachtreatment stage is expected to move a patient's teeth by a predeterminedamount. Teeth that deviate from the treatment plan may have moved lessthan anticipated or more than anticipated. If one or more of thepatient's teeth are not moved by the predetermined amount, this may becaused by complications such as roots of adjacent teeth colliding. Atblock 540, processing logic determines that the deviation exceeds adeviation threshold.

At block 545, processing logic determines one or more treatment planalterations based on the deviation. For example, there may be multipledifferent treatment paths to adjust a patient's teeth to the desiredfinal positions. If a first treatment path is not adjusting the teeth asexpected, one or more viable alternative treatment paths may bedetermined. Additionally, some types of tooth movement such asparticular rotations of teeth may not be achieved successfully withoutadding attachments to the teeth to be rotated. If a tooth has notundergone a desired rotation, then a suggested attachment may bedetermined for that tooth to apply additional rotational forces on thetooth. Other types of treatment plan alterations may also be determined.

In one embodiment, processing logic provides 3-D controls for a dentalpractitioner wearing the AR display to adjust the treatment plan. Forexample, the dental practitioner may move a tooth, rotate a tooth, etc.by interacting with a virtual 3-D model of the patient's dental archthat is displayed via the AR display.

At block 550, processing logic may determine additional teeth that haveundergone motion in accordance with the orthodontic treatment plan andthat therefore have the target positions and orientations. At block 555,processing logic generates a visual overlay comprising an indication ofan area of interest associated with the tooth. The visual overlay mayalso include a suggested treatment plan alteration. For example, thevisual overlay may indicate one or more regions on a tooth or teeth toplace attachments. The visual overlay may also include indications ofadditional teeth that have target positions and orientations asindicated in the orthodontic treatment plan. In an example, teeth thathave failed to move as indicated in the orthodontic treatment plan maybe highlighted in a first color while teeth that have moved according tothe orthodontic treatment plan may be highlighted in a second color.

The visual overlay may also indicate a desired position and orientationfor a tooth that has not moved as predicted. Accordingly, a dentalpractitioner may visually see the difference between one or more teethat the current treatment stage and the one or more teeth as they werepredicted to be at the current treatment stage. Processing logic mayprovide the dental practitioner with one or more options for showingplayback of the dental arch from initial positions of the teeth tocurrent positions of the teeth and all the way to treatment outcomepositions of the teeth. Such playback may be implemented as a stream ofvisual overlays that are projected onto the AR display. The visualoverlays may be superimposed over the actual patient's teeth in a fieldof view of the dental practitioner as though the patient's teeth aremoving according to the treatment plan before the dental practitioner'seyes. Processing logic may update a treatment outcome determinationbased on differences in the predicted tooth positions at the currenttreatment stage and the actual tooth positions at the current treatmentstage.

At block 560, processing logic outputs the visual overlay to a displayof the augmented reality device.

FIG. 6 illustrates a flow diagram for a method 600 of augmenting theview of a patient's mouth through an augmented reality display based ona clinical history of the patient, in accordance with an embodiment. Atblock 610 of method 600, processing logic receives image data of thedental arch generated by an image capture device of an augmented realitydevice. At block 615, processing logic computes teeth segmentation andidentifies gums from the received image data. After teeth segmentation,processing logic may determine the identities of individual teeth. Forexample, in dentistry each tooth is generally identified by a toothnumber (e.g., an American Dental Association tooth identifier, Palmernotation tooth identifier, or Federation Dentaire Internationale toothidentifier). Processing logic may determine the tooth number associatedwith each tooth identified in the received image data.

At block 620, processing logic determines a clinical history for thepatient that identifies a tooth and/or gum area. For example, theclinical history may identify that one or more dental procedures hadbeen performed on a particular tooth or gum area at a previous date. Theclinical history may also indicate previous dental conditions for thepatient such as broken teeth, swollen gums, gum recession, fillings, andso on. The clinical history may identify which teeth and/or location onthe dental arch the previous dental conditions applied to.

At block 625, processing logic determines a position of an area ofinterest associated with an identified tooth and or gum area from theclinical history. At block 630, processing logic generates a visualoverlay that includes information about a tooth and/or gum area from theclinical history at the identified area of interest. For example,processing logic may highlight tooth number five in the visual overlaywith an indicator that tooth number five has a crack if the clinicalhistory indicates such. Processing logic may additionally add to thevisual overlay a label for each of the one or more teeth that arevisible in the image data, where the label indicated an identity of atooth (e.g., identifies tooth 5). At block 635, processing logic outputsthe visual display to a display of an augmented reality device. Thevisual overlay is superimposed over a dental practitioner's view of thedental arch on the display at the position of the area of interest.

FIG. 7 illustrates a flow diagram for a method 700 of augmenting theview of a patient's mouth through an augmented reality display, inaccordance with an embodiment. Beginning in block 710, processing logicreceives image data of a dental arch. For example, the AR system mayreceive the image data from an image capture device. The image data mayinclude a two dimensional image or video data. In some embodiments, thereceived image data may include a three dimensional image orstereoscopic image data generated from a variety of image capturedevices.

In block 710, the processing logic may detect a feature in the image ofthe dental arch. For example, a feature may be a tooth, several teeth, agum line, a contour, or any other feature of the dental arch in theimage data. In some embodiments, the AR system may first identify ageneral feature such as a mouth or a dental arch, and then identify aspecific feature such as a tooth or gum line within the dental arch. Insome embodiments, the AR system may use previously identified featuresto aid detecting a new feature. For example, if a tooth was previouslyanalyzed from a previous image, the AR system may search for a newfeature representing a second tooth by searching image data near theoriginal tooth.

In block 730, the processing logic matches the feature to a portion of athree dimensional model. For example, the processing logic may generatetwo dimensional projections for the three dimensional model fromdifferent perspectives to simulate the image data that would be receivedfrom different camera angles and positions. The three dimensional modelmay be manipulated by the processing logic over six degrees of freedomof movement. The AR system may then use such projections to attempt tomatch the detected feature to a portion of the three dimensional model.In some embodiments, the processing logic may constrain the search for amatch between the detected feature and the three dimensional model tocertain potential image capture device positions. For example, theprocessing logic may limit the search to portions of the threedimensional model that are visible from an opening of a patient's mouthfor matches to the detected feature, or may search within a restrictedrange of positions expected to be viewed from an image capture device.

In some embodiments, processing logic may also compare the feature inthe image to a plurality of recorded features in a data store. Forexample, the data store may include reference data from previous imagesor scans taken for other patients. The recorded features in the datastore may be images of the features, models of the features, or anotherrepresentation of the features. The recorded features may be stored withan indication of clinical data relevant to the feature. For example, therecorded features may have metadata or a file associated with them thatstores AOIs associated with the features or an indication if a featureis normal and has no AOIs. In some embodiments, the processing logic maylimit the comparison of the detected feature to recorded features of thesame type. For example, the processing logic may compare a detectedtooth to recorded teeth, a detected gum-line to a recorded gum line, orthe like.

In block 740, the processing logic determines a perspective of the imagecapture device based on the matched feature. For example, if thedetected feature matches a portion of the three dimensional model whenrotated and placed in a specific position, the AR system may determine aperspective or position of an image capture device that generated theimage.

In block 750, the processing logic may determine a position of an AOI inthe image data based on the perspective of the image capture device. Theposition may be determined relative to the position of the image capturedevice, relative to the three dimensional model, or relative to thereceived image data. Because a feature of the dental arch in the imagedata is matched to the three dimensional model, and a position of theimage capture device is determined based on the match, determining aposition of an AOI relative to the image data, dental arch, or positionof the camera generates a position that can be correlated to the others.In some embodiments, the processing logic may determine a position ofthe AOI in a two dimensional field of view of the received image data.This position may then be converted to specific positions in the view ofeach eye of a dental practitioner using the AR system. For example, afirst position may be determined for the line of sight of a right eye,and a second position may be determined for the line of sight of theleft eye.

In block 760, the processing logic overlays an indication of the AOI onan AR display. The indication overlay may be positioned on the ARdisplay so as to appear in the position of the AOI on the dental arch.For example, the AR system may project an indication of the AOI ontoeach lens in the frame of a set of AR glasses. The indication may markthe AOI with a color or other indicator to highlight the AOI for thedental practitioner. The indicator may also show the type of AOI, orother information about the AOI.

FIGS. 8A and 8B are example illustrations of a portion of a view of anAR display having overlay indicators generated using methods describedwith reference to FIGS. 5-7, or similar methods. FIG. 8A is an exampleillustration of a portion of an AR display 830 showing indicators 835,840 positioned on the AR display to indicate movement in the position ofteeth in a dental arch compared to previous patient data. A visualoverlay for the AR display may be generated beginning with image data810. The image data 810 shows a current image of a patient's smile asseen by a dental practitioner through the AR display. The image data 810in FIG. 8A is shown as a two dimensional image, but may also be used byan AR system as stereoscopic image data, scan data, or other image data.

Previous image data 820 is also used by the AR system to determinewhether there has been a change in the dentition of the patient since aprevious scan. Similar to the image data 810, the image data 820 may bea two dimensional image, stereoscopic image data, previous scan data, athree dimensional model, or another representation of previous patientdata. The AR system may compare the image data 810 to the previous imagedata 820 to determine if there has been a change in the dental arch, orto identify other AOIs. In the example shown in FIG. 8A, the alignmentand position of the front teeth have changed since the previous data wastaken. Accordingly the AR system may determine a direction and amount ofchange in the position of the teeth. The change in the position of theteeth may then be translated into an indication for the dentalpractitioner.

A portion of a view of the AR display is shown in FIG. 8A having avisual overlay that includes the indications 835, 840 of the directionand amount of change of teeth in image data. Specifically, the view ofthe dental practitioner as shown in the image data is overlay with thevisual overlay showing the indicators 835, 840. The indications 835, 840may include a two dimensional (or in some embodiments three dimensional)vector indicating a direction and amount of movement of a particulartooth. A first color, graphic or line type may be used to show a vectorthat represents an amount of movement that was planned (e.g., forindicator 840) and a second color, graphic or line type may be used toshow a vector that represents an unplanned amount of movement, such asan amount of movement that is below a planned amount of movement (e.g.,for indicator 835). In some embodiments, other indicators may be used toconvey the movement of teeth in a dental arch. For example, if themovement is part of a planned orthodontic treatment, the indicators mayhighlight teeth based on whether their movement is on track with theplanned treatment. Accordingly, teeth that are in an expected positionmay be highlighted with one color, and teeth that are not moving asexpected may be highlighted with another color. In some embodiments, theAR system may indicate to the dental practitioner the teeth that are notmoving in an expected manner. In other embodiments, any other indicationof the position or movement of teeth may be generated and displayed inan AR display as alternatives or additions to the indicators 835, 840shown in FIG. 8A.

FIG. 8B is an example of a portion of a view 850 of an AR displayshowing an overlay of the skeletal structure of a patient. The skeletalstructure may be generated from a previous scan (e.g., a CT scan),previous x-ray imaging, or other previous data for the patient. The ARsystem may align the skeletal structure to the patient using methodssimilar to those discussed with reference to FIGS. 2-7. For example, theAR system may detect a feature in imaging data received from an imagecapture device. When looking at the front of the patient, such a featuremay be a tooth, a gum line, a contour of the dental arch, or anotherfeature in the imaging data. In some embodiments, the AR system may alsodetermine a feature from a profile view of a patient based on the dentalarch or based on other features, such as a patient's nose, chin, lips,or other structure. The AR system may then compare the feature to athree dimensional model (such as generated from previous scan, x-ray, orother data) and/or to optical image data to determine a position of theimage capture device relative to the virtual three dimensional modeland/or optical image data. The AR system may then provide an overlay ofthe patient's jaw as shown in the portion of the view 850 of an ARdisplay.

In some embodiments, the AR system may display only a portion of the jawof a patient as an overlay on the patient's face. For example, the ARsystem may display only a portion of the jaw that has an AOI for thepatient, or a portion of the jaw that is not visible from the viewpointof the AR system. The AR system may overlay portions of the jaw that arecovered by the lips or cheeks of the patient, for instance. In someembodiments, the dental practitioner may request the overlay of theskeletal structure of a patient's jaw or a type of overlay while viewingthe patient. For example, the dental practitioner may request to seex-ray images as an overlay on the patient's jaw, CT scan data as anoverlay, or a three dimensional model of the patient's jaw as anoverlay. The AR system may the project the overlay onto an AR display toaugment the view of the dental practitioner through the AR display.

FIG. 9 illustrates a flow diagram for a method 900 of determining areasof interest by an augmented reality device, in accordance with anembodiment. At block 910 of method 900, processing logic receives imagedata of a dental arch (e.g., of an oral cavity that includes a dentalarch). At block 920, processing logic processes the image data using aplurality of detection rules. Each detection rule may be or include adifferent dental condition profile, model and/or algorithm. The dentalcondition profile may include training data that was used to generate afeature set and or a dental condition model. Each dental condition rule(e.g., dental condition profile, model and/or algorithm) may be used todetect one or more dental conditions.

At block 930, processing logic determines a dental condition for thedental arch based on the processing performed at block 920. At block935, processing logic determines a position of an area of interest onthe dental arch. The area of interest is associated with the dentalcondition. At block 940, processing logic generates a visual overlaycomprising an indication of the dental condition at the position of thearea of interest. The indication of the dental condition may be, forexample, a contour of at least a portion of a tooth and/or gum thatincludes the dental condition. At block 950, processing logic outputsthe visual overlay to a display of the augmented reality device. Thevisual overlay is superimposed over a view of the dental arch on thedisplay at the position of the area of interest. Accordingly, a dentalpractitioner may see a real-world view of the patient's dental archalong with a computer generated overlay of the AOI.

FIG. 10 illustrates a flow diagram for a method 1000 of processing imagedata of a dental arch from an augmented reality device based on machinelearning profiles of dental conditions, in accordance with anembodiment. At block 1010 of method 1000, processing logic receivesimage data of a dental arch. At block 1020, processing logic processesthe image data using a dental condition profile generated based on atraining data set including positive example images of dental archesthat have a dental condition and negative example images of dentalarches that lack the dental condition. At block 1030, processing logicdetermines, for the image data, a degree of match to the positiveexample images and to the negative example images using a feature setand a dental condition model included in the dental condition profile.

At block 1032, processing logic determines whether the dental arch inthe image data has a dental condition associated with the dentalcondition profile. The dental arch is determined to have the dentalcondition if a tooth or gum area of the dental arch is determined tohave features that match features of the positive example images with aconfidence level that is greater than a confidence level threshold.Alternatively the dental arch is determined not to have the dentalcondition if there are no teeth or gum areas on the dental arch thathave features that are determined to match the positive example images.If at block 1032 it is determined that the dental arch does include thedental condition, then the method continues to block 1035. Otherwise themethod proceeds to block 1040.

At block 1035, processing logic determines an area of interest in theimage data that comprises the dental condition. At block 1040,processing logic generates a visual overlay comprising an indication ofthe dental condition at the position of the area of interest. At block1050, processing logic outputs the visual overlay to a display of theaugmented reality device. The visual overlay is superimposed over a viewof the dental arch on the display at the position of the area ofinterest.

FIG. 11 illustrates a flow diagram for a method 1100 of processing imagedata of a dental arch from an image capture device of an augmentedreality device to identify tooth wear, in accordance with an embodiment.The processes in FIG. 11 may be used in conjunction or as an alternativeto portions of the processes described with reference to FIGS. 2-10 todetect AOIs for teeth. Beginning in block 1105, an AR system receivesimage data of a dental arch. For example, the AR system may receive theimage data from an image capture device. The image data may include atwo dimensional image or video data. In some embodiments, the receivedimage data may include stereoscopic image data generated from a varietyof image capture devices. Furthermore, the image data may be a modelgenerated from stereoscopic image data.

The process may continue in blocks 1110 and 1120 to provide two methodsof detecting tooth wear. In some embodiments, the AR system may use onlythe first detection method or only the second detection method. Startingwith the first detection method in block 1110, the AR system computesteeth segmentation. To compute tooth segmentation, the AR system mayidentify outlines of teeth and identify regions of image data that arewithin the outlines to define as particular teeth. In particular, thetooth segmentation may be performed to identify the surface of a tooththat would contact another tooth. The segmentation computation may alsogenerate a model representing each tooth or a surface of each tooth.

In block 1115, the AR system continues to compare each tooth to modelsof crown shapes. For example, reference data in a data store may havemodels of various crown shapes. The crown shapes may be based on imagedata or based on virtual models of crowns. The models may include idealcrown shapes and/or crown shapes that have evident tooth wear. Based onthe comparison, the AR system may continue to block 1130 to identifytooth wear as an AOI. The AR system may then generate an indication ofthe tooth wear to display on the AR display.

In the second detection method, the AR system may begin in block 1120 byseparating teeth from gums in the image data. The separated teeth may beroughly separated based on the color of the teeth compared to the colorof the gums. In some embodiments, the teeth may be separated from thegums in another manner. The AR system may continue in block 1125 tocompare the tooth surfaces of the separated teeth against models ofhealthy teeth. The comparison may indicate portions of the teeth in theimage data that do not conform to an expected size, shape, or structurefrom the healthy models geometries. Based on the comparison, the ARsystem may continue to block 1130 to identify tooth wear as an AOI. TheAR system may then generate an indication of the tooth wear to displayon the AR display.

FIGS. 12A and 12B illustrate example outputs to an AR display based onlive analysis of image data of a dental arch. In FIG. 12A, tooth wear isindicated on a patient's dental arch. In some embodiments, the toothwear may have been detected according to methods described withreference to FIGS. 2-10, or based on other methods. FIG. 12A shows aportion 1210 of the view of an AR system before the AR system hasanalyzed image data from a dental arch. The AR system may then performprocessing on the image data to identify tooth wear. The tooth wear maythen be highlighted on a portion 1220 of the view of the AR display asshown. The indicators 1225 may identify the tooth wear for the dentalpractitioner. The indicators 1225 shown in FIG. 12A highlight specificareas of tooth wear in a color that contrasts with the tooth color. Insome embodiments, the AR system may select the color based on the AOIbeing tooth wear and another color could indicate another type of AOI.Additionally, other types of indicators may be used by the AR system.For example, the AR system may have indicators that point to, circle, orotherwise identify areas with tooth wear without or in addition tohighlighting the tooth wear. Furthermore, in some embodiments, the ARsystem may provide an audio indication of the AOI. For example, the ARsystem may have a speaker to announces the AOI (e.g., by tooth number)and the type of AOI. In some embodiments, the AR system may provide avariety of indications for a particular AOI.

FIG. 12B illustrates an example view of an AR display identifying AOIsof a dental arch. FIG. 12B shows a portion 1230 of a view of an ARdisplay with a visual overlay showing indicators of AOIs identifiedbased on image data of a dental arch. For example, the AOIs 1235 and1240 identified in the dental arch may be identified based on methodsdescribed with reference to FIGS. 2-7 and FIGS. 9-14 above, or based onother methods. As illustrated in FIG. 12B, the AR display shows thedental arch with an overlay 1235 and 1240 indicating areas of interest.

The first AOI 1235 is a dashed circle pointing out an area of interestand the second OAI 1240 is a solid circle pointing out an area ofinterest. In some embodiments, the different indicators may indicatedifferent types of AOIs. For example AOI 1235 may indicate plaque, whileAOI 1240 may indicate a potential cavity or crack. In some embodiments,the different indicators may indicate a severity of an identified AOI.For example, AOI 1235 may indicate a small area of plaque, while AOI1240 may indicate a large area of plaque. In some embodiments, theindicators may be of different shapes or styles to indicate AOIs ofdifferent types or severities. In addition, in some embodiments, the ARdisplay may further provide a text indicator describing what an overlayindicates.

FIG. 13 illustrates a flow diagram for a method 1300 of enhancing a viewof a patient's mouth as viewed through an augmented reality display, inaccordance with an embodiment. At block 1305 of method 1300, processinglogic receives image data including a patient's mouth (e.g., oralcavity) and a dental arch in the mouth or oral cavity. At block 1310,processing logic identifies an area in the image data associated with amouth/oral cavity comprising the dental arch. At block 1315, processinglogic determines a light enhancement to increase the visibility of themouth/oral cavity. At block 1320, processing logic generates a visualoverlay comprising the light enhancement. At block 1325, processinglogic outputs the visual overlay to a display of an augmented realitydevice.

FIG. 14 illustrates a flow diagram for a method 1400 of providing avisual overlay of a patient's mouth during a dental procedure to augmentthe dental procedure, in accordance with an embodiment. The procedure ortreatment performed by a dental practitioner may be a prosthodontic(restorative) or orthodontic procedure. Procedures may also be simplyperforming a scan of a patients jaw, taking x-rays of a patient's jaw,taking other imaging data, cleaning a patient's teeth, or the like.While particular treatments and procedures are described herein, theseare given as examples and the disclosure contemplates similar methodsand processes for any dental treatments or procedures.

At block 1410 of method 1400, processing logic receives image data of adental arch. For example, processing logic of an AR system may receivethe image data from an image capture device. The image data may includea two dimensional image or video data. In some embodiments, the receivedimage data may include stereoscopic image data generated from a varietyof image capture devices.

At block 1420, processing logic determines an intraoral procedure ortreatment to be performed on a tooth of the dental arch. At block 1430,processing logic detects a first area of interest in the image of thedental arch, wherein the first area of interest is associated with theintraoral procedure. The first area of interest may be determined usingany of the aforementioned techniques, such as those described withreference to FIGS. 2-11. At block 1435, processing logic provides avisual overlay for output on a display of an augmented reality deviceidentifying the first area of interest. The indicator for the AOI may beshown at a position of the AOI in a real-world view of the patient'sdental arch as seen by a dental practitioner, and may identify the typeof AOI and/or or other information about the AOI. In some embodiments,the AOI may be an area where a treatment or procedure is to beperformed. The indicator for the AOI presented in the visual overlay mayindicate the next step on the treatment or procedure, a location ortarget of a treatment or procedure, an indication of progress of atreatment or procedure, and so on. The visual overlay may be generatedusing any of the aforementioned techniques.

At block 1438, processing logic receives updated image data of thedental arch. The updated image data may be received in the same formatas the image data received in block 1410. For example, if stereoscopicimages were received from image capture devices in block 1410, the sameimage capture device may provide the same image data format in block1438. In some embodiments, different formats of image data may bereceived. For example, if the dental practitioner is partially or fullyobscuring a portion of the view of the dental arch, the AR system mayreceive image data from a subset of image capture devices or fromsecondary image devices that were not previously used. Furthermore, insome embodiments, the update image data may include a CT scan, x-rayimage data, or other image data than was received in block 1410.

At block 1440, processing logic determines whether the first area ofinterest is changed based on the updated image data and/or a comparisonof the updated image data to the previously received image data. Thefirst area of interest may change, for example, if the dental procedureis adding material to the dental arch or removing material from thedental arch. For example, if the dental procedure involves grinding oneor more teeth, then the first area of interest may change if the toothbeing ground corresponds to an area of interest. If the first area ofinterest has not changed than the method returns to block 1440. If thefirst area of interest has changed, then the method proceeds to block1445.

At block 1445, processing logic identifies a change to the first area ofinterest during the intraoral procedure. At block 1450, processing logicdetermines an update to the visual overlay based on the change to thefirst area of interest. At block 1455, processing logic provides theupdated visual overlay to the AR display. In an example, if the dentalprocedure is grinding a tooth, then the first area of interesthighlighted in the visual overlay may show an area of the tooth to beground in a color such as red. An area of the tooth that is not to beground may be shown in another color such as green. As the tooth isground, the visual overlay may be updated to reflect the material thathas been removed from the tooth. Additionally, if any portion of thetooth has been ground down to a predetermined finish area, then thatportion of the area of interest may be shown in a contrasting color.This may enable a dental practitioner to more accurately grind down atooth according to a treatment plan.

At block 1460, processing logic determines whether the dental procedurehas been completed. The processing logic may determine the procedure iscomplete based on a detected change to the image data, based on feedbackfrom the dental practitioner, based on a time change, or based on othercriteria or indication. If the procedure or treatment is not complete,the processing logic may return to block 1438 and continue to receiveupdated image data and to provide updated overlay indicators based onthe changes. If the procedure or treatment is determined to be complete,the AR system may end the method 1400. To end the method, the processinglogic may indicate that the procedure or treatment is complete, removeindicators regarding the AOI, move on to a next procedure or treatment,stop updating the overlay, or perform other tasks.

The method 1400 described with reference to FIG. 14 may be used toimplement a variety of dental procedures and treatments. As anon-limiting set of examples, the treatment or procedures discussed withreference to FIG. 14 may include placing attachments, interproximalreduction, computer tomography (CT) or x-ray scanning, cavity mapping,intraoral scanning, placement of a hole for implant, drilling, grinding,or any other dental treatment of procedure. In one example, the method1400 described with reference to FIG. 14 may include a live update to anoverlay of material to remove during an interproximal reductionprocedure. The AR system may show an initial map of material to removeduring the interproximal reduction.

FIG. 15A illustrates an example portion 1510 of a view of an AR displayshowing a visual overlay with an indication 1515 of an amount of toothto remove in an interproximal region between two teeth. The amount oftooth to remove may be determined based on a three dimensional model ofthe patient's dental arch as well as planned steps in an orthodontictreatment plan. As the dental practitioner removes material from thetooth, the AR system may receive additional updated image data of thedental arch. The AR system may then determine an amount of material thatis still to be removed. Based on the amount of material and position ofmaterial to remove, the AR system may update the visual overlay toprovide an indicator of a new amount of material to remove. In addition,the AR system may change a color of the overlay or provide an indicationwhen the planned amount of material has been removed from the tooth. Forexample, FIG. 15B illustrates an example portion 1520 of a view of an ARdisplay with an updated overlay based on the material removed from thetooth. The AR display in FIG. 15B includes an indication 1525 of anamount of material to remove and an indication of the amount of materialthat has been removed. In some embodiments, the indicator 1525 mayinstead change color gradually as material is removed. For example, theindicator 1515 may be completely green and gradually change colors untilthe planned amount of material has been removed. Then the indicator mayturn red to indicate that the procedure is complete.

In another example, the method 1400 described with reference to FIG. 14may include an update during x-ray scanning or intraoral scanning. Forexample, the AR system may determine areas with completed scanned dataand overlay those areas with the scanned data. For example, the ARsystem may display to the dental practitioner live updates to scanneddata as an overlay of the scanned data on the patient's dental arch. Insome embodiments, the AR system may use highlighting instead of scanneddata as an overlay to indicate either areas that have not been scannedor areas that have already been scanned.

In another example, the AR system may perform a method 1400 as describedwith reference to FIG. 14 to indicate to a dental practitioner anindication of placement of an attachment or ideal placement of a holefor an implant. For example, the AR system may access a threedimensional model of the patient's dentition and use that model toidentify an ideal placement of an attachment or a hole for an implant.The AR system may identify the placement based on automated analysis ofthe three dimensional model, or based on planned positions indicated onthe three dimensional model. FIG. 16 depicts an example of a portion1610 of an AR display showing an indicator 1615 of a placement of a holefor an implant to a dental practitioner. The indicator 1615 shows aplacement of a hole and a direction that the hole should be drilled.Similar indications may be provided for placement of an attachment. Forexample, the AR system may place an outline of the position of theattachment on a tooth. The AR system may then update the color of theoutline based on analysis of image data to recognize placement of theattachment.

FIG. 17A is an example of a portion 1710 of an AR display showing a liveocclusion map for a patient. In some embodiments, the AR system maygenerate an occlusion map based on analysis of contacts between thelower and upper jaw portions of a patient. For example, the occlusionmap may be generated from three dimensional models captured during acurrent session or from a previous treatment, procedure or scan. The ARsystem may then match the occlusion map to a feature detected in imagedata received by an image capture device. In some embodiments, the ARsystem may also update an occlusion map during a treatment or procedureby a dental practitioner. For example, if a dental practitioneridentifies an issue with an occlusion map (e.g., disruptive contacts)the dental practitioner may determine that grinding a portion of one ormore contacts may improve the patient's bite. Accordingly, the dentalpractitioner may begin grinding a portion of a contact to improve theoverall points of a patient's bite. The AR system may receive updatedimage data from the image capture device and determine a change in theshape of one or more teeth where the dental practitioner performedgrinding. The AR system may then calculate or receive a new occlusionmap based on the updated shape of teeth. The AR system may then overlaythe updated occlusion map on the dental arch of a patient on the ARdisplay to show changes to contacts. The dental practitioner may thendetermine when to stop grinding based on updates to the occlusion map.In some embodiments, the AR system may also indicate to the dentalpractitioner when to stop grinding based on a planned outcome for theocclusion map or the occlusion map meeting a particular threshold forplacement of contacts.

FIG. 17B illustrates a patient's upper dental arch 1730 and lower dentalarch 1720 from current image data with a visual overlay showing actualtooth movement compared to target tooth movement from a treatment plan.Teeth that have moved according to the treatment plan may be shown inthe visual overlay with a first color, line type or fill type and teeththat have not moved according to the treatment plan may be shown with asecond color, line type or fill type. Alternatively, portions of theteeth that are outside of a target region may be shown with the firstcolor, while portions of the teeth that are within a target region maybe shown with a second color.

FIG. 18 illustrates a flow diagram for a method 1800 of determiningareas of interest by an augmented reality device, in accordance with anembodiment. Method 1800 is a process for generating overlay informationfor display on an AR display during a treatment or procedure performedby a dental practitioner using a dental instrument. The procedure ortreatment performed by a dental practitioner may be prosthodontic(restorative) or orthodontic procedures. Procedures may also beperforming a scan of a patients jaw, taking x-rays of a patient's jaw,taking other imaging data, cleaning a patient's teeth, placing animplant, grinding a patient's tooth, or the like. While particulartreatments and procedures are described herein, these are given asexamples and the disclosure contemplates similar methods and processesfor any dental treatments or procedures.

Beginning in block 1810 an AR system receives image data of a dentalarch. For example, the AR system may receive the image data from animage capture device.

In block 1820, the AR system identifies one or more areas of interestbased on a detected feature. For example, the AR system may identifyareas of interest according to methods as described above with referenceto FIGS. 2-11. In some embodiments, areas of interest may be identifiedusing other methods or processes. The AR system may store the areas ofinterest with an indicator of a type of area of interest and position ofthe area of interest. For instance, coordinates in a three dimensionalmodel may be used to store a position of the area of interest.

In block 1830, the AR system overlays an indication of the AOI on an ARdisplay. The indication overlay may be positioned on the AR display soas to appear in the position of the AOI on the dental arch as viewed bya dental practitioner wearing the AR display. The indication may markthe AOI with a color or other indicator to highlight the AOI for thedental practitioner. The indicator may also show the type of AOI, orother information about the AOI. In some embodiments the AOI may be anarea where a treatment or procedure is to be performed. The indicatorpresented on the AR display may indicate the next step on the treatmentor procedure, a location or target of a treatment or procedure, or anindication of progress of a treatment or procedure.

In block 1840, the AR system receives updated image data of the dentalarch. The image data may be received in the same format as the imagedata received in block 1810. For example, if stereoscopic images werereceived from image capture devices in block 1810, the same imagecapture device may provide the same image data format in block 1810. Insome embodiments, different formats of image data may be received. Forexample, if the dental practitioner is partially or fully obscuring aportion of the view of the dental arch, the AR system may receive imagedata from a subset of image capture devices or from secondary imagedevices that were not previously used. Furthermore, in some embodiments,the update image data may include CT scan, x-ray image data, or otherimage data than was received in block 1810.

The AR system may then determine an update based on the updated imagedata and determine a position of a dental instrument used during theprocedure or treatment. In the first aspect, in block 1850, The ARsystem determines an update to the overlay on the display of the ARsystem. For example, the AR system may compare the updated image data topreviously received image data. Based on the comparison, the AR systemmay identify an update to an AOI. The AR system may then determine anupdate to the indicator of the AOI and provide that update to the ARdisplay so that the dental practitioner is presented with the updatedoverlay.

The AR system may also continue in block 1845 to determine a position ofa dental instrument relative to the AOI. The AR system may determine theposition of the dental instrument based on a sensor in the dentalinstrument and/or based on a position of the dental instrument in theupdated image data received by the AR system. The position of the dentalinstrument may be detected with six degrees of freedom to indicate aposition and rotation within three dimensional space. In someembodiments, the AR system may also correlate the position of the dentalinstrument to a planned position for treatment or procedures performedby the dental practitioner.

The AR system may then continue in block 1855 to generate an indicationof the position of the dental instrument relative to the tooth positionand/or a recommended or planned position of the dental instrument. Forexample, in some embodiments, the AR system may generate an indicationof a preferred position of a drill during an implant operation. Theindication may provide an indication to the dental practitioner whetherthe instrument is in a proper position.

In block 1860, the AR system may provide an indication of a change tothe area of interest and/or the position of the dental instrument on anAR display. The indication overlay may be positioned on the AR displayso as to appear in the position of the AOI on the dental arch. Forexample, the AR system may project an indication of the change to theAOI or to the position of the dental instrument onto each lens in theframe of a set of AR glasses. The indication may mark the AOI with acolor or other indicator to highlight the AOI for the dentalpractitioner. The indicator may also show the type of AOI, or otherinformation about the AOI.

The AR system may continue in block 1870 to determine whether thetreatment or procedure is complete. The AR system may determine theprocedure is complete based on a detected change to the image data,based on feedback from the dental practitioner, based on a time change,or based on other criteria or indication. If the procedure or treatmentis not complete, the AR system may continue to receive updated imagedata and to provide updated overlay indicators based on the changes. Ifthe procedure or treatment is determined to be complete, the AR systemmay end the method 1800. To end the method, the AR system may indicatethat the procedure or treatment is complete, remove indicators regardingthe AOI, move on to a next procedure or treatment, stop updating theoverlay, or perform other tasks.

The method 1800 described with reference to FIG. 18 may be used toimplement a variety of dental procedures and treatments. As anon-limiting set of examples, the treatment or procedures discussed withreference to FIG. 18 may include placing attachments, interproximalreduction, CT or x-ray scanning, intraoral scanning, cavity mapping,placement of a hole for implant, drilling, grinding, or any other dentaltreatment of procedure.

In one example implementation, the AR system may perform operations asdescribed with reference to FIG. 18 to provide control or providefeedback to operations of a dental drill during a treatment or procedureperformed by a dental practitioner. In some embodiments, the drill usedby a dental practitioner may include a sensor to determine the positionand orientation of the dental drill in relation to the AR system. Insome embodiments, the drill may include a visual marker to indicate theorientation of the dental instrument. For example, a OR code, a barcode,or another visual indicator may be indicated by a visual marker on thedental drill.

The AR system may then use the orientation and position of a dentaldrill in relation to an identified AOI to determine whether the drill isin a position relative to the AOI to perform the appropriate treatmentor procedure. If the dental drill is not in the correct position ororientation, the AR system may display on an AR display that theposition is not in the correct position. In some embodiments, the ARsystem may issue instructions to the dental drill to stop the dentaldrill from operating until the dental drill is in the correct positionand orientation.

In some embodiments, the AR system may also mark adjacent undercuts orimplant sites for particular procedures. Furthermore, the AR system mayalso indicate particular areas that may have cavities, tooth wear, orother issues that are being addressed by the current procedure ortreatment. Moreover, if the dental drill has reached a particular depthas calculated by the AR system, the AR system may cause the drill tostop operating or to stop operating until an override instruction isissued by a dental practitioner.

FIG. 19 illustrates a flow diagram for a method 1900 of providing avisual overlay in a view on an augmented reality display that providesinformation about a procedure to grind a tooth, in accordance with anembodiment. At block 1910 of method 1900, processing logic receivesimage data of a dental arch from an image capture device of an augmentedreality device. At block 1920, processing logic determines an intraoralprocedure to grind down at least one tooth to correct a malocclusion orfor another purpose. At block 1930, processing logic determines aportion of a tooth to be ground. At block 1932, processing logic maydetermine an occlusion map. At block 1935, processing logic provides avisual overlay for output on the display of the augmented realitydevice. The visual overlay identifies the portion of the tooth to beground and/or may identify the occlusion map.

At block 1937, processing logic receives updated image data of thedental arch. At block 1940, processing logic determines whether anyportion of the tooth has been ground based on processing the updatedimage data. If no portion of the tooth has been ground, then theoperation of block 1940 is repeated. If the tooth has been ground, thenthe method proceeds to block 1945. At block 1945, processing logicdetermines the remaining portion of the tooth to be ground. At block1950, processing logic may determine an updated occlusion map based onthe portion of the tooth that has been ground. The updated occlusion mapmay be determined by adjusting a three-dimensional model that includesan upper arch, a lower arch, and the contact points of teeth between theupper arch and lower arch. At block 1955, processing logic provides anupdated visual overlay.

At block 1960, processing logic determines whether the dental procedurehas been completed. The processing logic may determine the procedure iscomplete based on a detected change to the image data, based on feedbackfrom the dental practitioner, based on a time change, or based on othercriteria or indication. If the procedure or treatment is not complete,the processing logic may return to block 1937 and continue to receiveupdated image data and to provide updated overlay indicators based onthe changes. If the procedure or treatment is determined to be complete,the AR system may end the method 1900. To end the method, the processinglogic may indicate that the procedure or treatment is complete, removeindicators regarding the AOI, move on to a next procedure or treatment,stop updating the overlay, or perform other tasks.

FIG. 20 illustrates a flow diagram for a method 2000 of providing avisual overlay in an image on an augmented reality display that providesinformation that augments use of a dental tool (also referred to hereinas a dental instrument), in accordance with an embodiment. At block 2010of method 2000, processing logic receives image data of a dental archfrom an image capture device of an augmented reality device. Thereceived image data includes an image of a dental tool being used by adental practitioner to perform an intraoral procedure. At block 2020,processing logic determines an intraoral procedure for at least onetooth, where the intraoral procedure uses the dental tool.

At block 2030, processing logic determines a desired position andorientation of the dental tool. At block 2035, processing logic providesa visual overlay for output on the display of an augmented realitydevice identifying the desired position and orientation. At block 2037,processing logic receives updated image data, where the updated imagedata includes a current position and orientation of the dental tool.

At block 2040, processing logic determines based on the updated imagedata whether the dental tool has the desired position and orientation.If the dental tool does not have the desired position and orientation,then the method returns to block 2040. If the dental tool does have thedesired position and orientation, then the method proceeds to block2045.

At block 2045, processing logic may activate the dental tool and/oradjust one or more settings of the dental tool. Additionally, processinglogic may output a command to a haptic device such as haptic gloves or ahaptic module of the dental tool to cause the haptic device to provide ahaptic feedback to the dental practitioner to indicate that the dentaltool has reached the desired position and orientation.

At block 2055, processing logic determines an update to the desiredposition and orientation of the dental tool. At block 2058, processinglogic provides an updated visual overlay for output on the display ofthe augmented reality device identifying the desired position andorientation. At block 2060, processing logic determines whether thedental procedure has been completed. If the procedure or treatment isnot complete, the processing logic may return to block 2037 and continueto receive updated image data and to provide updated overlay indicatorsbased on the changes. If the procedure or treatment is determined to becomplete, the method may end.

FIG. 21 illustrates a flow diagram for a method 2100 of facilitatingplacement of attachments on a patient's teeth using an augmented realitydevice, in accordance with an embodiment. At block 2110 of method 2100,processing logic receives image data of a dental arch from an augmentedreality devices' image capture device. At block 2120, processing logicreceives a treatment plan. The treatment plan may be, for example, anorthodontic treatment plan that indicates forces to be applied to teethat various stages of treatment. Some types of forces may be improved byadding attachments to teeth. For example, attachments may improve somerotational forces on teeth.

At block 2130, processing logic determines a type of attachment and/orlocation for the attachment from the treatment plan. At block 2135,processing logic determines an area in the image data of the dental archcorresponding to locations for the attachments. At block 2140,processing logic generates a visual overlay comprising an indication ofthe areas to place the dental attachments. At block 2150, processinglogic outputs the visual overlay to a display of the augmented realitydevice. The visual overlay may be superimposed over a view of the dentalarch as viewed by the dental practitioner on the display at the positionof the area of interest. Accordingly, the dental practitioner may see inan augmented reality display an indication of where to place anattachment.

FIG. 22 illustrates a flow diagram for a method 2200 of facilitating anintraoral scan session using an augmented reality device, in accordancewith an embodiment. At block 2210 of method 2200, processing logicreceives image data of a dental arch from an image capture device of anaugmented reality device. At block 2220, processing logic receives aplurality of images from an intraoral scanner scanning the dental arch.At block 2230, processing logic registers the image from the intraoralscanner. This may include stitching images from the intraoral scannertogether to build a model of the dental arch.

At block 2235, processing logic determines an area of the dental arch inthe image data that has been scanned by the intraoral scanner and/orthat has not been scanned. At block 2240, processing logic generates avisual overlay for output on the display of an augmented reality displayidentifying an area that has been scanned by the intraoral scanner usinga first visual indication (e.g., a first color) and/or identifying anarea that has not been scanned by the intraoral scanner using a secondvisual indication (e.g., a second color). Processing logic mayadditionally or alternatively perform an analysis of the scanned regionsfrom the intraoral scan data to identify any dental conditions. AOIsidentifying these dental conditions may then be determined and shown inthe visual overlay. This enables a dental practitioner to immediatelysee any possible dental conditions during an intraoral scan session.

At block 2260, processing logic determines whether additional imageshave been received from the intraoral scanner. If additional images havebeen received from the intraoral scanner, the method returns to block2230, and those additional images are registered to the previousintraoral images generated by the intraoral scanner. If no additionalimages are received, then the method continues to block 2265.

At block 2265, processing logic determines whether there are any areasof the dental arch that have not been scanned. For example, a dentalpractitioner may inadvertently skip over certain portions or regions ofthe dental arch during a scanning session. Such areas may be highlightedin the visual overlay. Accordingly, processing logic is capable ofquickly identifying any holes in the image data (and thus the virtual3-D model) of the dental arch. If there are no un-scanned areas of thedental arch, then the method proceeds to block 2280. If there areun-scanned areas of the dental arch, then the method continues to block2270.

At block 2270, processing logic determines whether the image data fromthe augmented reality device's image capture device is sufficient tofill in gaps associated with un-scanned areas. For example, image datafor an un-scanned area that is small and that is bordered by scannedareas on both sides may be provided based on image data from theaugmented reality device's image capture device. However, if the viewrepresented in the received image data is low quality or blocked by lipsor other obstructions, or the un-scanned area is larger than a thresholdsize, then the received image data may be insufficient to fill in thegaps. If the received image data can be used to fill in the gaps, thenthe method continues to block 2280. If the image data cannot be used tofill the gaps, the method continues to block 2275.

At block 2275, processing logic generates a notification for output onthe augmented reality display. The notification may indicate an area ofinterest that shows the un-scanned area. At block 2280, processing logicgenerates a three-dimensional model of the dental arch using the imagesfrom the intraoral scanner. Additionally, processing logic may use thereceived image data in addition to the data from the intraoral scannerto generate the three-dimensional model of the dental arch if there weresmall un-scanned areas that could be filled in using the image data.

FIG. 23 illustrates a flow diagram for a method 2300 of using anaugmented reality display for an intraoral scanner, in accordance withan embodiment. At block 2310 of method 2300, processing logic receivesimage data of a dental arch from an image capture device of an augmentedreality display. At block 2320, processing logic receives a plurality ofintraoral images from an intraoral scanner scanning the dental arch.

At block 2330, processing logic registers the intraoral images togetherand stitches the intraoral images together based on the registration. Inone embodiment, processing logic performs image registration for eachpair of adjacent or overlapping intraoral images (e.g., each successiveframe of an intraoral video). Image registration algorithms are carriedout to register two adjacent intraoral images, which essentiallyinvolves determination of the transformations which align one image withthe other. Image registration may involve identifying multiple points ineach image (e.g., point clouds) of an image pair, surface fitting to thepoints of each image, and using local searches around points to matchpoints of the two adjacent images. For example, processing logic maymatch points of one image with the closest points interpolated on thesurface of the other image, and iteratively minimize the distancebetween matched points. Processing logic may also find the best match ofcurvature features at points of one image with curvature features atpoints interpolated on the surface of the other image, withoutiteration. Processing logic may also find the best match of spin-imagepoint features at points of one image with spin-image point features atpoints interpolated on the surface of the other image, withoutiteration. Other techniques that may be used for image registrationinclude those based on determining point-to-point correspondences usingother features and minimization of point-to-surface distances, forexample. Other image registration techniques may also be used.

Many image registration algorithms perform the fitting of a surface tothe points in adjacent images, which can be done in numerous ways.Parametric surfaces such as Bezier and B-Spline surfaces are mostcommon, although others may be used. A single surface patch may be fitto all points of an image, or alternatively, separate surface patchesmay be fit to any number of a subset of points of the image. Separatesurface patches may be fit to have common boundaries or they may be fitto overlap. Surfaces or surface patches may be fit to interpolatemultiple points by using a control-point net having the same number ofpoints as a grid of points being fit, or the surface may approximate thepoints by using a control-point net which has fewer number of controlpoints than the grid of points being fit. Various matching techniquesmay also be employed by the image registration algorithms.

In one embodiment, processing logic may determine a point match betweenimages, which may take the form of a two dimensional (2D) curvaturearray. A local search for a matching point feature in a correspondingsurface patch of an adjacent image is carried out by computing featuresat points sampled in a region surrounding the parametrically similarpoint. Once corresponding point sets are determined between surfacepatches of the two images, determination of the transformation betweenthe two sets of corresponding points in two coordinate frames can besolved. Essentially, an image registration algorithm may compute atransformation between two adjacent images that will minimize thedistances between points on one surface, and the closest points to themfound in the interpolated region on the other image surface used as areference.

Processing logic may repeat image registration for all adjacent imagepairs of a sequence of intraoral images to obtain a transformationbetween each pair of images, to register each image with the previousone. At block 2335, processing logic then integrates all images into asingle virtual 3D model of the dental arch being scanned by applying theappropriate determined transformations to each of the images. Eachtransformation may include rotations about one to three axes andtranslations within one to three planes.

At block 2338, processing logic determines from the image data a regionin a view from a wearer of the AR display that is outside of the dentalarch (and outside of an oral cavity that includes the dental arch). Forexample, if the dental practitioner is looking at a patient while he orshe is performing an intraoral scan procedure, his field of view mightalso include the chair on which the patient is sitting, a portion of aroom, and so on. At block 2340, processing logic generates a visualoverlay for output on the AR display that includes the virtual 3-D modelgenerated based on the received intraoral images. The virtual 3-D modelmay be a partial model of the patient's dental arch based on intraoralimages so far received.

At block 2345, processing logic sends the visual overlay to the ARdisplay worn by the dental practitioner. Additionally, processing logicmay send the visual overlay to a VR display worn by the patient. The ARdisplay displays the visual overlay such that the virtual 3-D model ofthe dental arch is shown in the region of the view for the dentalpractitioner that is outside of the dental arch (and oral cavity). Thatway the virtual 3-D model does not obstruct a view of the patient's oralcavity. The dental practitioner may interact with the virtual 3-D modelusing controls on the intraoral scanner (e.g., a touch interface on theintraoral scanner) or other input mechanisms such as motion controls.For example, the dental practitioner may wear haptic gloves, use ahaptic wand, or use another haptic device. The user may “touch” thevirtual 3-D model with the haptic device, which may cause a forcefeedback when the user “touches” the 3-D model. The user may interactwith the virtual 3-D model to rotate the virtual 3-D model, zoom in orout on the virtual 3-D model, reposition the virtual 3-D model in thedental practitioner's field of view, and so forth. Based on the userinput, processing logic may generate a new virtual overlay showing thevirtual 3-D model with the new orientation, new zoom setting, newposition in the dental practitioner's field of view, and so on.

At block 2360, processing logic determines whether any additionalintraoral images have been received from the intraoral scanner. If newintraoral images are received, the method returns to block 2330, and thenew intraoral images are registered and stitched together with theprevious intraoral images. The virtual 3-D model is then updated toincorporate the new image data. Accordingly, the virtual 3-D model maygrow and become more complete as the patient's dental arch is scanned.At any time the dental practitioner may refer to the virtual 3-D modelin his field of view to determine whether there are any issues that needto be addressed, whether there are any regions that should be rescannedor that have not been scanned, and so on.

If at block 2360 no additional intraoral images are received, and thedental practitioner indicates that the scan is complete, the methodproceeds to block 2380. At block 2380, processing logic generates avirtual 3-D model of the dental arch from the intraoral images. Thisvirtual 3-D model may be a more accurate and detailed virtual 3-D modelthan the one generated at block 2335. Similar algorithms may be used togenerate both virtual 3-D models, but more iterations may be performedto refine the virtual 3-D model at block 2380, more processor resourcesmay be used, and more time may be used to generate the final virtual 3-Dmodel. The operations of blocks 2320-2360 may be performed during a scanmode of an intraoral scan application. The operations of block 2380 maybe performed during a processing mode of the intraoral scan application.

FIG. 24A illustrates a flow diagram for another method 2400 of using anaugmented reality display for an intraoral scanner, in accordance withan embodiment. At block 2410 of method 2400, processing logic determinesa current stage or mode of an intraoral scanning procedure. At block2420, processing logic determines menu options for the current stage ormode of the intraoral scan procedure. At block 2440, processing logicgenerates a visual overlay for output on an AR display, where the visualoverlay includes a display of the menu options. The display of the menuoptions may be a 2-D display, a 3-D display, or a combination of 2-Ddisplay elements and 3-D display elements. The menu options may include,for example, a menu bar with different drop down menus, icons, and/orother graphical representations. Processing logic may place the menu ofoptions in a fixed position on the virtual overlay, so that the menu ispresented on an AR display worn by the dental practitioner at the samelocation in the dental practitioner's field of view regardless of whatthe menu might occlude from the dental practitioner's field of view.Alternatively, processing logic may receive image data from an imagecapture device of the AR display worn by the dental practitioner, andmay determine whether the menu would obstruct the view of a patient. Theposition of the menu may then be adjusted so that it does not obstructthe view of the patient. For example, the menu may be repositioned froma top of the AR display to a side of the AR display.

At block 2445, processing logic sends the visual overlay to the ARdisplay. At block 2450, processing logic receives an input selecting amenu option from the menu. The dental practitioner may use buttons, atouch input, or other input mechanism (e.g., a gyroscope and/oraccelerometer that act as a motion input) from an intraoral scanner toselect a menu option. The dental practitioner may also provide voiceinput to select the menu option.

At block 2452, processing logic updates the visual overlay based on theselected menu option. For example, the selected menu option may cause anintraoral scan application to change modes or stages (e.g., between aplanning mode, a scan mode, a processing mode, and a transmission mode).The selected menu option may also cause additional menu options to bedisplayed (e.g., by expanding a drop down menu). The updated data isreflected in the update to the visual overlay. At block 2456, processinglogic sends the visual overlay update to the AR display.

At block 2460, processing logic determines whether a new stage or modeof the intraoral scan procedure has been reached. If a new stage hasbeen reached (e.g., based on user input selecting a next intraoral scanmode), the method returns to block 2420 and new menu options associatedwith the new stage or mode are determined. Otherwise the method proceedsto block 2465.

At block 2465, processing logic determines whether the intraoral scanprocedure is complete. If the intraoral scan procedure is not complete,the method returns to block 2452. If the intraoral scan procedure iscomplete, the method ends.

Methods 2300 and 2400 may be used together to provide a virtual displayfor an intraoral scan application that is used in conjunction with anintraoral scanner to scan a patient's dental arches. The virtual displaythat is projected onto an AR display (e.g., an AR headset or AR goggles)has numerous advantages over a standard display shown on a computerscreen. For example, the virtual display may appear much larger than astandard display. Additionally, the dental practitioner can view andinteract with the virtual display without looking away from the patient.The virtual display can also be positioned anywhere in the field of viewof the dental practitioner, such as over the patient's head, to the sideof the patient's head, or wherever is convenient for the dentalpractitioner.

FIG. 24B illustrates a virtual display 2470 for an intraoral scanapplication that is displayed on an AR display, in accordance with anembodiment. As shown, a dental practitioner 2482 is wearing an ARdisplay 2480 while using an intraoral scanner 2478 to scan an oralcavity of a patient 2476. The AR display 2482 displays the virtualdisplay 2470 so that it appears to float over a head of the patient2476. The virtual display 2470 for the intraoral scan applicationincludes a menu 2472 that includes multiple menu options (e.g., asdiscussed with reference to method 2400) as well as a virtual 3-D modelgenerated based on the intraoral scan (e.g., as discussed with referenceto method 2300).

FIG. 25A illustrates a flow diagram for a method 2500 of using anaugmented reality display and an intraoral scanner to provide a zoomedin view of a dental procedure, in accordance with an embodiment. Atblock 2510 of method 2500, processing logic receives image data of adental arch from an image capture device of an AR display. At block2520, processing logic receives an intraoral image of a dental tool andan area of the dental arch proximate to the dental tool. In oneembodiment, the dental arch of the patient and the dental tool are firstscanned using the intraoral scanner to generate virtual 3-D models ofthe dental arch and the dental tool. This may facilitate identifyingregions of the dental arch and portions of the dental tool in laterreceived images.

At block 2530, processing logic generates a zoomed in view of the dentaltool and area of the dental arch proximate to the dental tool from theintraoral image. In one embodiment, processing logic determines an AOIfrom the intraoral image, and generates the zoomed in view (alsoreferred to as an enlarged image or magnified image) of just the AOI.The AOI may include, for example, a region of the dental arch beingoperated on and a dental tool that is operating on the region of thedental arch. If 3-D models of the dental tool and the dental arch havepreviously been generated, this may enable processing logic to morequickly and easily determine a position and orientation of the dentaltool relative to a particular tooth being operated on. Tracking accuracycan be on the order of 20-50 microns in embodiments.

At block 2538, processing logic determines from the image data receivedfrom the image capture device of the AR display a region in a view of awearer of the AR display that is outside of the dental arch (and outsideof an oral cavity). At block 2540, processing logic generates a visualoverlay for output on the AR display that includes the zoomed in view.At block 2545, processing logic sends the visual overlay to the ARdisplay. The AR display displays the visual overlay such that the zoomedin view is shown in the region of the wearer's field of view that isoutside of the dental arch (and oral cavity). Thus, the zoomed in viewof the dental procedure does not occlude a dental practitioner's actualreal-world view of the dental procedure. In an example, a dentalpractitioner wearing the AR display may see both his patient and anenlarged image of the region in the patient's oral cavity where thedental practitioner is currently operating floating in the air above thepatient. The enlarged image may include one or several teeth and adental tool (e.g., a drill) being used. Processing logic may also sendthe visual overlay to a VR display worn by a patient. This enables thepatient to also view the dental procedure as it is performed.

At block 2560, processing logic determines whether additional imageshave been received from the intraoral scanner and/or the AR display. Inone embodiment, the intraoral scanner and the AR display each generate astream of image data (e.g., a live video feed). Accordingly, additionalimages may be received from both the intraoral scanner and the ARdisplay throughout the dental procedure. This enables the zoomed in view(enlarged image) of the teeth being operated on and the dental toolbeing used to be updated in real time so that it is in sync with themovements of the dental tool and progress of the dental procedure. Thedental practitioner may determine how to manipulate the dental tooleither by looking directly in the mouth of the patient (e.g., as thedental practitioner could do without wearing an AR display) or bylooking at the enlarged image or zoomed in view displayed in the ARdisplay. The large unobscured image (zoomed in view) of the area beingoperated on may make tooth manipulation and operation on the toothsignificantly easier. For example, even slight motions of the dentaltool can be identified in the magnified (enlarged) image. This enablesslight modifications to be easily identified. Errors may beautomatically detected and signaled as well. For example, undercuts orexcessively large preparations that do not leave enough space for a newcrown or bridge may be identified during the intraoral procedure. If noadditional image data is received, and the dental procedure iscompleted, then the method ends.

In some embodiments, the image capture device of the AR display maycapture images using different wavelengths. For example, the imagecapture device may generate infrared images, which may show data aboutthe inside of a tooth being operated on (since teeth are transparent tolight at the near-infrared spectrum).

In some embodiments, the images from the image capture device of the ARdisplay and/or the images from the intraoral scanner generated duringthe intraoral procedure are recorded. These recorded images may act as a“black box recorder” that documents the actions of the dentalpractitioner so that someone can later learn from the dental procedureor identify what went wrong during the dental procedure.

FIG. 25B illustrates a dental practitioner 2580 operating on a patient2581, in accordance with an embodiment. As shown, the dentalpractitioner 2580 is wearing an AR display 2584. The dental practitioner2580 is using a dental tool 2588 to operate on a tooth of the patient2581. An assistant is holding an intraoral scanner 2586 that is directedso as to image the dental procedure. Based on the images from theintraoral scanner, an enlarged image 2582 of the tooth being operated onand the dental tool 2588 performing the operation is generated and sentto the AR display 2584. The AR display 2584 displays the enlarged image2582 so that it appears to be floating over a head of the patient 2581.

FIG. 26 illustrates a flow diagram for a method 2600 of generating amodel for a dental arch from images captured by an image capture deviceassociated with an augmented reality display, in accordance with anembodiment. At block 2610 of method 2600, processing logic receives astream of images of a patient's face from an image capture device. Theimage capture device may be an image capture device of an AR displayworn by a dental practitioner.

At block 2620, processing logic processes the stream of images.Processing the stream of images may include processing the stream ofimages using one or more machine learning profiles that have beentrained to identify particular types of images. For example, a firstmachine learning profile may be trained to identify images thatrepresent a left and/or right profile view of a patient's face. A secondmachine learning profile may be trained to identify images thatrepresent a front view of a patient's face. An additional machinelearning profile may be trained to identify images that represent amouth that is maximally open, and so on. The machine learning profilesmay be generated using the machine learning profile generationtechniques described herein above. Alternatively, image analysisprofiles or models may be generated manually (without using machinelearning) that analyze images to determine specific properties of theimages. For example, an image analysis profile may be generated thatsearches for a nose tip and a nose base, that computes a vector betweenthe nose tip and nose base, and that computes a horizontal component ofthe vector. In one embodiment, an image analysis profile may begenerated that identifies most prominent facial features in the images(e.g., protruding nose, lips, chin, etc.). Those images that include themost prominent facial features may be selected using such an imageanalysis profile.

At block 2630, processing logic may determine a subset of images fromthe stream of images that satisfy image selection criteria. At block2638, processing logic selects the determined subset. The image analysisprofiles or models may be used to process each incoming image, and thenfrom the images received so far select an image that is a best match fora particular type of image. For example, image analysis profiles ormodels may select a best left and right profile view from the imagereceived so far. Then if a better left or right profile view is laterreceived, that previously selected left or right profile view image maybe discarded and replaced by the new left or right profile view. Theprocess may continue until no new images are received.

Examples of images that may be selected for the determined subset mayinclude an image representing a left profile of the face in which amouth of the patient is closed, an image representing a right profile ofthe face in which the mouth of the patient is closed, an imagerepresenting the left profile of the face in which the mouth of thepatient is open, an image representing the right profile of the face inwhich the mouth of the patient is open, an image representing a frontview of the face in which the mouth of the patient is closed, and animage representing the front view of the face in which the mouth of thepatient is open. Other examples of images that may be selected for thedetermined subset may include an image in which a lower jaw of thepatient has been moved left relative to an upper jaw of the patient, animage in which the lower jaw of the patient has been moved rightrelative to the upper jaw of the patient, and an image in which themouth is maximally open.

In one embodiment, an image analysis profile for identifying bestprofile images is used to analyze images. For each image, processinglogic may perform image recognition on the image to identify a tip of anose and a base of the nose. Alternatively, the processing logic mayidentify a tip of the nose and one or more other facial features.Processing logic may then compute a vector between the tip of the noseand the base of the nose (or between the tip of the nose and the one ormore other facial features). Processing logic may then determine adirection and magnitude of a horizontal component of the vector. Avector having a first direction may be identified as a left profileimage and a vector having a second direction may be identified as aright profile image. Processing logic may compare, between each of theplurality of images, the direction and the magnitude for the horizontalcomponent of the vector. Processing logic may then select the imagehaving the maximum magnitude and a first direction for the horizontalportion of the vector as the left profile image and may select the imagehaving the maximum magnitude and a second direction for the horizontalportion of the vector as the right profile image.

In one embodiment, an image analysis profile for identifying best jawarticulation extreme images is used to analyze images. For each image,processing logic may perform image recognition on the image to identifyan upper jaw of a patient and to identify a lower jaw of the patient.Processing logic may additional determine a midline of the upper jaw byfinding a midline between the patient's ears, between the patient'seyes, between equal left and right halves of teeth on the upper jaw, orby another technique. An average of midlines computed from the eyes,nose, ears, teeth, cheek bones, and/or other facial features may also bedetermined. The midline is a vertical line that is at the middle of thepatient's face (e.g., that acts as an axis of symmetry between the leftand right side of the patient's face). The upper jaw midline is themidline for the upper dental arch and the lower jaw midline is for thelower dental arch.

Processing logic may additionally determine a midline of the lower jawof the patient. The lower jaw midline may be determined by determiningan axis of symmetry for the lower lip of the patient, for the lowerexposed teeth of the patient, for the lower jaw profile of the patient,or from other facial features of the lower jaw. Midlines of multipledifferent facial features of the lower jaw may additionally be averaged.Processing logic may then determine a horizontal distance between theupper jaw midline and the lower jaw midline.

Processing logic may compare, between each of the plurality of images,the horizontal distance between the first midline and the secondmidline. Processing logic may then select an image having a maximumhorizontal distance and a lower jaw that is to the right of the upperjaw. Processing logic may additionally select an image having a maximumhorizontal distance and a lower jaw that is to the left of the upperjaw.

At block 2640, processing logic stores the selected subset of images. Atblock 2645, processing logic generates one or more models of thepatient's jaw from the selected subset of images. The jaw model may be,for example, an articulation model of the patient's jaw. For example,processing logic may use multiple images of the patient's jaw that aretaken in multiple arch positions (e.g., max to the right, max to theleft, max protruding, and so on) to calculate intermediate jawpositions. The calculated positions may then be used to generate anarticulation model of the patient's jaw in different positions.Alternatively, or additionally, the selected images may be used todetermine left-right symmetry for a patient, to determine a smile linefor the patient, to determine facial proportions for the patient, togather a record of post treatment results, to gather a record ofpre-treatment conditions, to record progress of an orthodontictreatment, and so on.

At block 2660, processing logic determines whether additional imageshave been received. If additional images have been received, the methodreturns to block 2630 and the additional images are processed and thencompared to the previously selected subset of images. The new images maythen either be discarded or used to replace one or more of thepreviously selected images. For example, if a previous left profileimage was of a patient turning their head partway to the left and a newimage is of the patient turning their head all the way to the left, thenthe new image may replace the previously selected left profile image.The previously generated jaw models may then be updated based on thenewly selected image or images.

If no additional images are received at block 2660, then the method maycontinue to block 2680. At block 2680, processing logic generates anocclusion map for the patient based on the one or more jaw models and/orbased on a virtual 3-D model of an upper and lower arch of the patientgenerated from an intraoral scan of the upper and lower arches.

The occlusion plane is defined as the horizontal plane through the tipsof the buccal cusps of the premolars or the tips of the mesiobuccalcusps of the first molars and first premolars. In some embodiments, theocclusion plane is used as a reference plane for defining an X-Y-Z gridsystem used to generate the occlusion map. Cross-sections that arenormal to the occlusion plane and that go through a point on an uppertooth and a point on an opposite lower tooth may be used to determinedistances between surfaces of the upper and lower teeth at variousareas. Alternatively, or additionally, some cross-sections may be takenat a cross-sectional plane passing through the Z-axis and making anarbitrary angle with the Y axis.

Since the coordinates of all points comprising the virtual 3-D model ofan upper and lower arch are known, the distances between opposite pointson the grid line can easily be determined. Let the distance betweenpoint I′ on the surface of upper tooth and its “facing partner” point I″on the surface of lower tooth, be denoted by d(I′,I″), then

d(I′,I″)=|Z(I′)−Z(I″)|,

where Z(I′), Z(I″) are the Z coordinates of the points I′, I″,respectively. The absolute value of the difference between thecoordinates has been taken since only the magnitude of the difference isof interest.

In this manner the distances between the pairs of points may be found.Different distance values may be represented in the occlusion map usingdifferent colors. For example, distances of 0 (which denote contact),may be shown as red, small distances may be shown as orange, mediumdistances may be shown as yellow, and larger distances may be shown asblue. The above distances may then be represented by a map of coloreddots, or pixels, according to a particular color scheme (e.g., such asprovided in the above example). In other words, the values of thedistances between opposite pairs of points on opposite upper and lowerteeth are mapped onto colored pixels on a straight line with thedistance between adjacent pixels equal to the distance between adjacentgrid lines on which the adjacent pairs of opposite points correspondingto the adjacent pixels are situated.

The map of distances between pairs of opposite points on opposite teethis referred to as an “occlusion map” for the opposite pair of teeth. Itis often convenient to superimpose an occlusion map on the upper dentalarch and/or lower dental arch. The occlusion map could be superimposedon the outline of a view of the teeth of both the upper and lower arches(e.g., as viewed by a dental practitioner through and AR display). Thisaffords easy monitoring of dental procedures by allowing a dentalpractitioner to see the relationship between the surfaces of oppositeteeth (i.e. the distances between opposite pairs of points on oppositeteeth) with the jaws closed, by studying the dental occlusion map withrespect to both the upper and lower teeth. A tooth (or teeth) can befitted with a crown (or a bridge), or a tooth can be ground, and theinfluence of the change made on the relationship between opposite teethon the upper and lower jaws can be seen by noting color changes in thedental occlusion map overlaid on the view of the dental arches as seenby the dental practitioner through the AR display. Changes can continueto be made until a desired spatial relationship between opposite teethis achieved.

The jaw articulation model can be applied to the virtual 3-D model ofthe upper and lower arches to determine or refine occlusion maps fordifferent relative jaw positions between the upper and lower dentalarches. An average of the occlusion maps for the different relativepositions of the upper and lower dental arches may optionally becomputed to determine a final occlusion map. The average of theocclusion maps may be a weighted average, where more likely upper jaw tolower jaw relative positions (e.g., less extreme positions) are weightedmore heavily than less likely upper to lower jaw relative positions. Thejaw articulation model may additionally improve the identification offunctioning contacts (contacts between upper and lower teeth that are onthe inner region of the teeth) and interfering contacts (contactsbetween upper and lower teeth that are in the outer region of theteeth). The articulation model can show, for example, how contactbetween the upper and lower teeth has changed from tooth grinding.

FIG. 27 illustrates a flow diagram for a method 2700 of attaching audionotes (also referred to as voice notes) to image data from an imagecapture device associated with an augmented reality display, inaccordance with an embodiment. At block 2710 of method 2700, processinglogic receives a stream of images of a patient's face from an imagecapture device associated with an AR display. The AR display may be wornby a dental practitioner viewing a patient's face. At block 2720,processing logic receives an instruction to generate a note for acurrent image. The current image may be an image that represents a viewof the patient's face that the dental practitioner currently sees whenhe issues the command to generate the note. The command to generate thenote may be issued via a voice command, a gesture command, or a press ofa button on an input device.

At block 2730, processing logic receives an audio note pertinent to thecurrent image. For example, the audio note may be a diagnosis of adental condition visible in the current image, a reminder to examine orcheck up on a dental condition, and so on. At block 2738, processinglogic saves the current image and the audio note. The audio note may besaved as an audio file such as a way file, an mp3 file, an aac file, foror other audio file type. The current image may be saved as a jpg file,a bmp file, a png file, or other image file type. The image file and theaudio file may be linked (e.g., by storing them as being related in arelational database). Accordingly, when processing logic later receivesa request to access the image at block 2740, processing logic mayretrieve both the image and the associated audio note. Processing logicmay then display the image and in parallel play the audio note at block2745.

Method 2700 provides the advantage that a dental practitioner's handsmay remain free while taking the audio notes. The dental practitionercan initiate the note taking process, generate the voice note, generatethe image to associate with the voice note, and store the image andvoice note while the dental practitioner is working on a patient's oralcavity all without taking his hands away from the patient's oral cavity.Thus, notes can be generated without any burden to the dentalpractitioner and without taking additional time for the dentalpractitioner.

FIG. 28 illustrates selected images 2800 from a set of images generatedby an image capture device associated with an augmented reality display,in accordance with an embodiment. As shown, a plurality of images 2805are received. From the plurality of images 2805, a left profile imagewith mouth closed 2810, front view image with mouth closed 2815, rightprofile image with mouth closed 28220, image with mouth maximally open2825, left profile image with mouth open (smiling and showing teeth)2835, front view image with mouth open (smiling and showing teeth) 2840and right profile image with mouth open (smiling and showing teeth) 2845are selected. The remainder of the images 2805 may be discarded. Theseselected images may be used to generate a jaw model.

FIG. 29 illustrates additional selected images 2900 from a set of imagesgenerated by an image capture device associated with an augmentedreality display, in accordance with an embodiment. The selected imagesinclude a first image 2905 in which the patient is moving his lower jawto the right as much as possible, a second image 2910 in which thepatient is moving his lower jaw to the left as much as possible, and athird image 2915 in which the patient has opened his mouth as wide aspossible. These selected images may be used to generate a jaw model,such as an articulation model.

FIG. 30 illustrates a diagrammatic representation of a machine in theexample form of a computing device 3000 within which a set ofinstructions, for causing the machine to perform any one or more of themethodologies discussed herein, may be executed. In alternativeembodiments, the machine may be connected (e.g., networked) to othermachines in a Local Area Network (LAN), an intranet, an extranet, or theInternet. The machine may operate in the capacity of a server or aclient machine in a client-server network environment, or as a peermachine in a peer-to-peer (or distributed) network environment. Themachine may be a personal computer (PC), a tablet computer, a set-topbox (STB), a Personal Digital Assistant (PDA), a cellular telephone, aweb appliance, a server, a network router, switch or bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines (e.g., computers)that individually or jointly execute a set (or multiple sets) ofinstructions to perform any one or more of the methodologies discussedherein. In one embodiment, the computer device 3000 corresponds tocomputing devices 105 of FIG. 1.

The example computing device 3000 includes a processing device 3002, amain memory 3004 (e.g., read-only memory (ROM), flash memory, dynamicrandom access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), astatic memory 3006 (e.g., flash memory, static random access memory(SRAM), etc.), and a secondary memory (e.g., a data storage device3028), which communicate with each other via a bus 3008.

Processing device 3002 represents one or more general-purpose processorssuch as a microprocessor, central processing unit, or the like. Moreparticularly, the processing device 3002 may be a complex instructionset computing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,processor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processing device 3002may also be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. Processing device 3002 is configured to execute theprocessing logic (instructions 3026) for performing operations and stepsdiscussed herein.

The computing device 3000 may further include a network interface device3022 for communicating with a network 3064. The computing device 3000also may include a video display unit 3010 (e.g., a liquid crystaldisplay (LCD) or a cathode ray tube (CRT)), an alphanumeric input device3012 (e.g., a keyboard), a cursor control device 3014 (e.g., a mouse),and a signal generation device 3020 (e.g., a speaker).

The data storage device 3028 may include a machine-readable storagemedium (or more specifically a non-transitory computer-readable storagemedium) 3024 on which is stored one or more sets of instructions 3026embodying any one or more of the methodologies or functions describedherein, such as instructions for an AR processing module 3050. Anon-transitory storage medium refers to a storage medium other than acarrier wave. The instructions 3026 may also reside, completely or atleast partially, within the main memory 3004 and/or within theprocessing device 3002 during execution thereof by the computer device3000, the main memory 3004 and the processing device 3002 alsoconstituting computer-readable storage media.

The computer-readable storage medium 3024 may also be used to store anAR processing module 3050, which may correspond to the similarly namedcomponent of FIGS. 1A-1B. The computer readable storage medium 3024 mayalso store a software library containing methods for an AR processingmodule 3050. While the computer-readable storage medium 3024 is shown inan example embodiment to be a single medium, the term “computer-readablestorage medium” should be taken to include a single medium or multiplemedia (e.g., a centralized or distributed database, and/or associatedcaches and servers) that store the one or more sets of instructions. Theterm “computer-readable storage medium” shall also be taken to includeany medium other than a carrier wave that is capable of storing orencoding a set of instructions for execution by the machine and thatcause the machine to perform any one or more of the methodologies of thepresent invention. The term “computer-readable storage medium” shallaccordingly be taken to include, but not be limited to, solid-statememories, and optical and magnetic media.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other embodiments will beapparent upon reading and understanding the above description. Althoughembodiments of the present invention have been described with referenceto specific example embodiments, it will be recognized that theinvention is not limited to the embodiments described, but can bepracticed with modification and alteration within the spirit and scopeof the appended claims. Accordingly, the specification and drawings areto be regarded in an illustrative sense rather than a restrictive sense.The scope of the invention should, therefore, be determined withreference to the appended claims, along with the full scope ofequivalents to which such claims are entitled.

1. A system comprising: an intraoral scanner to generate a plurality ofintraoral images of a dental arch of a patient during an intraoralscanning procedure; an augmented reality (AR) display comprising animage capture device, the image capture device of the AR display togenerate additional image data representative of a view from a wearer ofthe AR display; and a computing device operatively coupled to theintraoral scanner and the AR display, the computing device to: receivethe plurality of intraoral images from the intraoral scanner; generate avirtual three-dimensional model of at least a portion of the dental archfrom the plurality of intraoral images during the intraoral scanningprocedure; receive the additional image data from the AR display;determine, from the additional image data, a region of the view that isoutside of the dental arch; generate a visual overlay comprising thevirtual three-dimensional model; and send the visual overlay to the ARdisplay, wherein the AR display is to display the visual overlay suchthat the virtual three-dimensional model is shown in the region of theview that is outside of the dental arch.
 2. The system of claim 1,wherein the computing device is further to: determine an area of thedental arch that has been scanned based on the plurality of intraoralimages; and add to the visual overlay at least one of a first visualindication of the area that has been scanned or a second visualindication of an additional area of the dental arch that has not beenscanned.
 3. The system of claim 2, wherein the computing device isfurther to: perform a diagnosis of the area of the dental arch that hasbeen scanned; and add an indication of the diagnosis to the visualoverlay.
 4. The system of claim 1, wherein the computing device isfurther to: determine a stage of the intraoral scanning procedure;determine a plurality of menu options for the stage of the intraoralscan procedure; and add a display of the plurality of menu options tothe visual overlay.
 5. The system of claim 1, wherein the computingdevice is further to: determine a location of the intraoral scanner andof an area of the dental arch proximate to the intraoral scanner in theadditional image data; generate a zoomed in view of the location of theintraoral scanner and of the area of the dental arch proximate to theintraoral scanner; and add the zoomed in view to the visual overlay,wherein the AR display is to display the visual overlay such that thezoomed in view is shown in the region of the view that is outside of thedental arch.
 6. The system of claim 1, further comprising: a virtualreality (VR) display to be worn by the patient; wherein the computingdevice is further to send at least one of the additional image data orthe visual overlay to the VR display.
 7. The system of claim 1, whereinthe plurality of intraoral images comprise images of a dental toolproximate to an area of the dental arch, and wherein the computingdevice is further to: generate a zoomed in view of the dental tool andof the area of the dental arch proximate to the dental tool from theplurality of intraoral images; and add the zoomed in view to the visualoverlay, wherein the AR display is to display the visual overlay suchthat the zoomed in view is shown in the region of the view that isoutside of the dental arch.
 8. A method comprising: receiving aplurality of intraoral images of a patient from an intraoral scannerduring an intraoral scanning procedure; generating, by a computingdevice, a virtual three-dimensional model of at least a portion of adental arch from the plurality of intraoral images during the intraoralscanning procedure; receiving additional image data from an augmentedreality (AR) display, wherein the additional image data isrepresentative of a view from a wearer of the AR display; determining,from the additional image data, a region of the view that is outside ofthe dental arch; generating, by the computing device, a visual overlaycomprising the virtual three-dimensional model; and sending the visualoverlay to the AR display, wherein the AR display is to display thevisual overlay such that the virtual three-dimensional model is shown inthe region of the view that is outside of the dental arch.
 9. The methodof claim 8, further comprising: determining an area of the dental archthat has been scanned based on the plurality of intraoral images; andadding to the visual overlay at least one of a first visual indicationof the area that has been scanned or a second visual indication of anadditional area of the dental arch that has not been scanned.
 10. Themethod of claim 9, further comprising: performing a diagnosis of thearea of the dental arch that has been scanned; and adding an indicationof the diagnosis to the visual overlay.
 11. The method of claim 8,further comprising: determining a stage of the intraoral scanningprocedure; determining a plurality of menu options for the stage of theintraoral scan procedure; and adding a display of the plurality of menuoptions to the visual overlay.
 12. The method of claim 8, furthercomprising: determining a location of the intraoral scanner and of anarea of the dental arch proximate to the intraoral scanner in theadditional image data; generating a zoomed in view of the location ofthe intraoral scanner and of the area of the dental arch proximate tothe intraoral scanner; and adding the zoomed in view to the visualoverlay, wherein the AR display is to display the visual overlay suchthat the zoomed in view is shown in the region of the view that isoutside of the dental arch.
 13. The method of claim 8, furthercomprising: sending at least one of the additional image data or thevisual overlay to a virtual reality (VR) display worn by the patient.14. The method of claim 8, wherein the plurality of intraoral imagescomprise images of a dental tool proximate to an area of the dentalarch, the method further comprising: generating a zoomed in view of thedental tool and of the area of the dental arch proximate to the dentaltool from the plurality of intraoral images; and adding the zoomed inview to the visual overlay, wherein the AR display is to display thevisual overlay such that the zoomed in view is shown in the region ofthe view that is outside of the dental arch.
 15. A non-transitorycomputer readable storage medium comprising instructions that, whenexecuted by a processing device, cause the processing device to performoperations comprising: receiving a plurality of intraoral images of apatient from an intraoral scanner during an intraoral scanningprocedure; generating, the processing device, a virtualthree-dimensional model of at least a portion of a dental arch from theplurality of intraoral images during the intraoral scanning procedure;receiving additional image data from an augmented reality (AR) display,wherein the additional image data is representative of a view from awearer of the AR display; determining, from the additional image data, aregion of the view that is outside of the dental arch; generating, bythe processing device, a visual overlay comprising the virtualthree-dimensional model; and sending the visual overlay to the ARdisplay, wherein the AR display is to display the visual overlay suchthat the virtual three-dimensional model is shown in the region of theview that is outside of the dental arch.
 16. The non-transitory computerreadable storage medium of claim 15, the operations further comprising:determining an area of the dental arch that has been scanned based onthe plurality of intraoral images; and adding to the visual overlay atleast one of a first visual indication of the area that has been scannedor a second visual indication of an additional area of the dental archthat has not been scanned.
 17. The non-transitory computer readablestorage medium of claim 16, the operations further comprising:performing a diagnosis of the area of the dental arch that has beenscanned; and adding an indication of the diagnosis to the visualoverlay.
 18. The non-transitory computer readable storage medium ofclaim 15, the operations further comprising: determining a stage of theintraoral scanning procedure; determining a plurality of menu optionsfor the stage of the intraoral scan procedure; and adding a display ofthe plurality of menu options to the visual overlay.
 19. Thenon-transitory computer readable storage medium of claim 15, theoperations further comprising: determining a location of the intraoralscanner and of an area of the dental arch proximate to the intraoralscanner in the additional image data; generating a zoomed in view of thelocation of the intraoral scanner and of the area of the dental archproximate to the intraoral scanner; and adding the zoomed in view to thevisual overlay, wherein the AR display is to display the visual overlaysuch that the zoomed in view is shown in the region of the view that isoutside of the dental arch.
 20. The non-transitory computer readablestorage medium of claim 15, wherein the plurality of intraoral imagescomprise images of a dental tool proximate to an area of the dentalarch, the operations further comprising: determining a location of thedental tool and of an area of the dental arch proximate to the dentaltool in the plurality of intraoral images; generating a zoomed in viewof the dental tool and of the area of the dental arch proximate to thedental tool from the plurality of intraoral images; and adding thezoomed in view to the visual overlay, wherein the AR display is todisplay the visual overlay such that the zoomed in view is shown in theregion of the view that is outside of the dental arch.